active_column 0.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/.gitignore +4 -0
- data/.rvmrc +7 -0
- data/Gemfile +4 -0
- data/Gemfile.lock +38 -0
- data/README.html +156 -0
- data/README.md +150 -0
- data/Rakefile +2 -0
- data/active_column.gemspec +26 -0
- data/lib/active_column.rb +9 -0
- data/lib/active_column/base.rb +90 -0
- data/lib/active_column/connection.rb +15 -0
- data/lib/active_column/version.rb +3 -0
- data/spec/active_column/base_crud_spec.rb +107 -0
- data/spec/active_column/base_finders_spec.rb +65 -0
- data/spec/spec_helper.rb +34 -0
- data/spec/support/aggregating_tweet.rb +10 -0
- data/spec/support/compound_key.rb +4 -0
- data/spec/support/config/storage-conf.xml +54 -0
- data/spec/support/simple_key.rb +4 -0
- data/spec/support/tweet.rb +6 -0
- data/spec/support/tweet_dm.rb +14 -0
- metadata +131 -0
data/.gitignore
ADDED
data/.rvmrc
ADDED
data/Gemfile
ADDED
data/Gemfile.lock
ADDED
@@ -0,0 +1,38 @@
|
|
1
|
+
PATH
|
2
|
+
remote: .
|
3
|
+
specs:
|
4
|
+
active_column (0.0.1)
|
5
|
+
simple_uuid
|
6
|
+
|
7
|
+
GEM
|
8
|
+
remote: http://rubygems.org/
|
9
|
+
specs:
|
10
|
+
cassandra (0.8.2)
|
11
|
+
json
|
12
|
+
rake
|
13
|
+
simple_uuid (>= 0.1.0)
|
14
|
+
thrift_client (>= 0.4.0)
|
15
|
+
diff-lcs (1.1.2)
|
16
|
+
json (1.4.6)
|
17
|
+
rake (0.8.7)
|
18
|
+
rspec (2.2.0)
|
19
|
+
rspec-core (~> 2.2)
|
20
|
+
rspec-expectations (~> 2.2)
|
21
|
+
rspec-mocks (~> 2.2)
|
22
|
+
rspec-core (2.2.1)
|
23
|
+
rspec-expectations (2.2.0)
|
24
|
+
diff-lcs (~> 1.1.2)
|
25
|
+
rspec-mocks (2.2.0)
|
26
|
+
simple_uuid (0.1.1)
|
27
|
+
thrift (0.2.0.4)
|
28
|
+
thrift_client (0.5.0)
|
29
|
+
thrift (~> 0.2.0)
|
30
|
+
|
31
|
+
PLATFORMS
|
32
|
+
ruby
|
33
|
+
|
34
|
+
DEPENDENCIES
|
35
|
+
active_column!
|
36
|
+
cassandra
|
37
|
+
rspec
|
38
|
+
simple_uuid
|
data/README.html
ADDED
@@ -0,0 +1,156 @@
|
|
1
|
+
<h1>ActiveColumn</h1>
|
2
|
+
|
3
|
+
<p>ActiveColumn is a framework for saving and retrieving data from Cassandra in a "time line" model. It is loosely based
|
4
|
+
on concepts in ActiveRecord, but is adapted to saving data in which rows in Cassandra grow indefinitely over time, such
|
5
|
+
as in the oft-used Twitter example for Cassandra.</p>
|
6
|
+
|
7
|
+
<h2>Installation</h2>
|
8
|
+
|
9
|
+
<p>Add ActiveColumn to your Gemfile:</p>
|
10
|
+
|
11
|
+
<pre>
|
12
|
+
gem 'active_column'
|
13
|
+
</pre>
|
14
|
+
|
15
|
+
<p>Install with bundler:</p>
|
16
|
+
|
17
|
+
<pre>
|
18
|
+
bundle install
|
19
|
+
</pre>
|
20
|
+
|
21
|
+
<h2>Usage</h2>
|
22
|
+
|
23
|
+
<h3>Configuration</h3>
|
24
|
+
|
25
|
+
<p>ActiveColumn requires the <a href="https://github.com/fauna/cassandra">cassandra gem</a>. You must provide ActiveColumn with an
|
26
|
+
instance of a Cassandra object. You can do this very simply like this:</p>
|
27
|
+
|
28
|
+
<pre>
|
29
|
+
ActiveColumn.connection = Cassandra.new('my_keyspace', '127.0.0.1:9160')
|
30
|
+
</pre>
|
31
|
+
|
32
|
+
<p>However, in a real app this is not flexible enough, so I often create a cassandra.yml file and configure Cassandra in an
|
33
|
+
initializer.</p>
|
34
|
+
|
35
|
+
<p>config/cassandra.yml</p>
|
36
|
+
|
37
|
+
<pre>
|
38
|
+
test:
|
39
|
+
home: ":"
|
40
|
+
servers: "127.0.0.1:9160"
|
41
|
+
keyspace: "myapp_test"
|
42
|
+
thrift:
|
43
|
+
timeout: 3
|
44
|
+
retries: 2
|
45
|
+
|
46
|
+
development:
|
47
|
+
home: ":"
|
48
|
+
servers: "127.0.0.1:9160"
|
49
|
+
keyspace: "myapp_development"
|
50
|
+
thrift:
|
51
|
+
timeout: 3
|
52
|
+
retries: 2
|
53
|
+
</pre>
|
54
|
+
|
55
|
+
<p>config/initializers/cassandra.rb</p>
|
56
|
+
|
57
|
+
<pre>
|
58
|
+
config = YAML.load_file(Rails.root.join("config", "cassandra.yml"))[Rails.env]
|
59
|
+
$cassandra = Cassandra.new(config['keyspace'],
|
60
|
+
config['servers'],
|
61
|
+
config['thrift'])
|
62
|
+
|
63
|
+
ActiveColumn.connection = $cassandra
|
64
|
+
</pre>
|
65
|
+
|
66
|
+
<p>As you can see, I create a global $cassandra variable, which I use in my tests to validate data directly in Cassandra.</p>
|
67
|
+
|
68
|
+
<p>One other thing to note is that you obviously must have Cassandra installed and running! Please take a look at the
|
69
|
+
<a href="https://github.com/carbonfive/mama_cass">mama_cass gem</a> for a quick way to get up and running with Cassandra for
|
70
|
+
development and testing.</p>
|
71
|
+
|
72
|
+
<h3>Saving data</h3>
|
73
|
+
|
74
|
+
<p>To make a model in to an ActiveColumn model, just extend ActiveColumn::Base, and provide two pieces of information:
|
75
|
+
* Column Family
|
76
|
+
* Function(s) to generate keys for your rows of data</p>
|
77
|
+
|
78
|
+
<p>The most basic form of using ActiveColumn looks like this:</p>
|
79
|
+
|
80
|
+
<pre>
|
81
|
+
class Tweet < ActiveColumn::Base
|
82
|
+
column_family :tweets
|
83
|
+
keys :user_id
|
84
|
+
end
|
85
|
+
</pre>
|
86
|
+
|
87
|
+
<p>Then in your app you can create and save a tweet like this:</p>
|
88
|
+
|
89
|
+
<pre>
|
90
|
+
tweet = Tweet.new( :user_id => 'mwynholds', :message => "I'm going for a bike ride" )
|
91
|
+
tweet.save
|
92
|
+
</pre>
|
93
|
+
|
94
|
+
<p>When you run #save, ActiveColumn saves a new column in the "tweets" column family in the row with key "mwynholds". The
|
95
|
+
content of the row is the Tweet instance JSON-encoded.</p>
|
96
|
+
|
97
|
+
<p><em>Key Generator Functions</em></p>
|
98
|
+
|
99
|
+
<p>This is great, but quite often you want to save the content in multiple rows for the sake of speedy lookups. This is
|
100
|
+
basically de-normalizing data, and is extremely common in Cassandra data. ActiveColumn lets you do this quite easily
|
101
|
+
by telling it the name of a function to use to generate the keys during a save. It works like this:</p>
|
102
|
+
|
103
|
+
<pre>
|
104
|
+
class Tweet < ActiveColumn::Base
|
105
|
+
column_family :tweets
|
106
|
+
keys :user_id => :generate_user_keys
|
107
|
+
|
108
|
+
def generate_user_keys
|
109
|
+
[ attributes[:user_id], 'all']
|
110
|
+
end
|
111
|
+
end
|
112
|
+
</pre>
|
113
|
+
|
114
|
+
<p>The code to save the tweet is the same as the previous example, but now it saves the tweet in both the "mwynholds" row
|
115
|
+
and the "all" row. This way, you can pull out the last 20 of all tweets quite easily (assuming you needed to do this
|
116
|
+
in your app).</p>
|
117
|
+
|
118
|
+
<p><em>Compound Keys</em></p>
|
119
|
+
|
120
|
+
<p>In some cases you may want to have your rows keyed by multiple values. ActiveColumn supports compound keys,
|
121
|
+
and looks like this:</p>
|
122
|
+
|
123
|
+
<pre>
|
124
|
+
class TweetDM < ActiveColumn::Base
|
125
|
+
column_family :tweet_dms
|
126
|
+
keys [ { :user_id => :generate_user_keys }, { :recipient_id => :recipient_ids } ]
|
127
|
+
|
128
|
+
def generate_user_keys
|
129
|
+
[ attributes[:user_id], 'all ]
|
130
|
+
end
|
131
|
+
end
|
132
|
+
</pre>
|
133
|
+
|
134
|
+
<p>Now, when you create a new TweetDM, it might look like this:</p>
|
135
|
+
|
136
|
+
<pre>
|
137
|
+
dm = TweetDM.new( :user_id => 'mwynholds', :recipient_ids => [ 'fsinatra', 'dmartin' ], :message => "Let's go to Vegas" )
|
138
|
+
</pre>
|
139
|
+
|
140
|
+
<p>This tweet direct message will saved to four different rows in the "tweet_dms" column family, under these keys:
|
141
|
+
* mwynholds:fsinatra
|
142
|
+
* mwynholds:dmartin
|
143
|
+
* all:fsinatra
|
144
|
+
* all:dmartin</p>
|
145
|
+
|
146
|
+
<p>Now my app can pretty easily figure find all DMs I sent to Old Blue Eyes, or to Dino, and it can also easily find all
|
147
|
+
DMs sent from <em>anyone</em> to Frank or Dino.</p>
|
148
|
+
|
149
|
+
<p>One thing to note about the TweetDM class above is that the "keys" configuration at the top looks a little uglier than
|
150
|
+
before. If you have a compound key and any of the keys have custom key generators, you need to pass in an array of
|
151
|
+
single-element hashes. This is in place to support Ruby 1.8, which does not have ordered hashes. Making sure the keys
|
152
|
+
are ordered is necessary to keep the compounds keys canonical (ie: deterministic).</p>
|
153
|
+
|
154
|
+
<h3>Finding data</h3>
|
155
|
+
|
156
|
+
<p>Working on this...</p>
|
data/README.md
ADDED
@@ -0,0 +1,150 @@
|
|
1
|
+
# ActiveColumn
|
2
|
+
|
3
|
+
ActiveColumn is a framework for saving and retrieving data from Cassandra in a "time line" model. It is loosely based
|
4
|
+
on concepts in ActiveRecord, but is adapted to saving data in which rows in Cassandra grow indefinitely over time, such
|
5
|
+
as in the oft-used Twitter example for Cassandra.
|
6
|
+
|
7
|
+
## Installation
|
8
|
+
|
9
|
+
Add ActiveColumn to your Gemfile:
|
10
|
+
<pre>
|
11
|
+
gem 'active_column'
|
12
|
+
</pre>
|
13
|
+
|
14
|
+
Install with bundler:
|
15
|
+
<pre>
|
16
|
+
bundle install
|
17
|
+
</pre>
|
18
|
+
|
19
|
+
## Usage
|
20
|
+
|
21
|
+
### Configuration
|
22
|
+
|
23
|
+
ActiveColumn requires the [cassandra gem](https://github.com/fauna/cassandra). You must provide ActiveColumn with an
|
24
|
+
instance of a Cassandra object. You can do this very simply like this:
|
25
|
+
|
26
|
+
<pre>
|
27
|
+
ActiveColumn.connection = Cassandra.new('my_keyspace', '127.0.0.1:9160')
|
28
|
+
</pre>
|
29
|
+
|
30
|
+
However, in a real app this is not flexible enough, so I often create a cassandra.yml file and configure Cassandra in an
|
31
|
+
initializer.
|
32
|
+
|
33
|
+
config/cassandra.yml
|
34
|
+
<pre>
|
35
|
+
test:
|
36
|
+
home: ":"
|
37
|
+
servers: "127.0.0.1:9160"
|
38
|
+
keyspace: "myapp_test"
|
39
|
+
thrift:
|
40
|
+
timeout: 3
|
41
|
+
retries: 2
|
42
|
+
|
43
|
+
development:
|
44
|
+
home: ":"
|
45
|
+
servers: "127.0.0.1:9160"
|
46
|
+
keyspace: "myapp_development"
|
47
|
+
thrift:
|
48
|
+
timeout: 3
|
49
|
+
retries: 2
|
50
|
+
</pre>
|
51
|
+
|
52
|
+
config/initializers/cassandra.rb
|
53
|
+
<pre>
|
54
|
+
config = YAML.load_file(Rails.root.join("config", "cassandra.yml"))[Rails.env]
|
55
|
+
$cassandra = Cassandra.new(config['keyspace'],
|
56
|
+
config['servers'],
|
57
|
+
config['thrift'])
|
58
|
+
|
59
|
+
ActiveColumn.connection = $cassandra
|
60
|
+
</pre>
|
61
|
+
|
62
|
+
As you can see, I create a global $cassandra variable, which I use in my tests to validate data directly in Cassandra.
|
63
|
+
|
64
|
+
One other thing to note is that you obviously must have Cassandra installed and running! Please take a look at the
|
65
|
+
[mama_cass gem](https://github.com/carbonfive/mama_cass) for a quick way to get up and running with Cassandra for
|
66
|
+
development and testing.
|
67
|
+
|
68
|
+
### Saving data
|
69
|
+
|
70
|
+
To make a model in to an ActiveColumn model, just extend ActiveColumn::Base, and provide two pieces of information:
|
71
|
+
* Column Family
|
72
|
+
* Function(s) to generate keys for your rows of data
|
73
|
+
|
74
|
+
The most basic form of using ActiveColumn looks like this:
|
75
|
+
<pre>
|
76
|
+
class Tweet < ActiveColumn::Base
|
77
|
+
column_family :tweets
|
78
|
+
keys :user_id
|
79
|
+
end
|
80
|
+
</pre>
|
81
|
+
|
82
|
+
Then in your app you can create and save a tweet like this:
|
83
|
+
<pre>
|
84
|
+
tweet = Tweet.new( :user_id => 'mwynholds', :message => "I'm going for a bike ride" )
|
85
|
+
tweet.save
|
86
|
+
</pre>
|
87
|
+
|
88
|
+
When you run #save, ActiveColumn saves a new column in the "tweets" column family in the row with key "mwynholds". The
|
89
|
+
content of the row is the Tweet instance JSON-encoded.
|
90
|
+
|
91
|
+
*Key Generator Functions*
|
92
|
+
|
93
|
+
This is great, but quite often you want to save the content in multiple rows for the sake of speedy lookups. This is
|
94
|
+
basically de-normalizing data, and is extremely common in Cassandra data. ActiveColumn lets you do this quite easily
|
95
|
+
by telling it the name of a function to use to generate the keys during a save. It works like this:
|
96
|
+
|
97
|
+
<pre>
|
98
|
+
class Tweet < ActiveColumn::Base
|
99
|
+
column_family :tweets
|
100
|
+
keys :user_id => :generate_user_keys
|
101
|
+
|
102
|
+
def generate_user_keys
|
103
|
+
[ attributes[:user_id], 'all']
|
104
|
+
end
|
105
|
+
end
|
106
|
+
</pre>
|
107
|
+
|
108
|
+
The code to save the tweet is the same as the previous example, but now it saves the tweet in both the "mwynholds" row
|
109
|
+
and the "all" row. This way, you can pull out the last 20 of all tweets quite easily (assuming you needed to do this
|
110
|
+
in your app).
|
111
|
+
|
112
|
+
*Compound Keys*
|
113
|
+
|
114
|
+
In some cases you may want to have your rows keyed by multiple values. ActiveColumn supports compound keys,
|
115
|
+
and looks like this:
|
116
|
+
|
117
|
+
<pre>
|
118
|
+
class TweetDM < ActiveColumn::Base
|
119
|
+
column_family :tweet_dms
|
120
|
+
keys [ { :user_id => :generate_user_keys }, { :recipient_id => :recipient_ids } ]
|
121
|
+
|
122
|
+
def generate_user_keys
|
123
|
+
[ attributes[:user_id], 'all ]
|
124
|
+
end
|
125
|
+
end
|
126
|
+
</pre>
|
127
|
+
|
128
|
+
Now, when you create a new TweetDM, it might look like this:
|
129
|
+
|
130
|
+
<pre>
|
131
|
+
dm = TweetDM.new( :user_id => 'mwynholds', :recipient_ids => [ 'fsinatra', 'dmartin' ], :message => "Let's go to Vegas" )
|
132
|
+
</pre>
|
133
|
+
|
134
|
+
This tweet direct message will saved to four different rows in the "tweet_dms" column family, under these keys:
|
135
|
+
* mwynholds:fsinatra
|
136
|
+
* mwynholds:dmartin
|
137
|
+
* all:fsinatra
|
138
|
+
* all:dmartin
|
139
|
+
|
140
|
+
Now my app can pretty easily figure find all DMs I sent to Old Blue Eyes, or to Dino, and it can also easily find all
|
141
|
+
DMs sent from *anyone* to Frank or Dino.
|
142
|
+
|
143
|
+
One thing to note about the TweetDM class above is that the "keys" configuration at the top looks a little uglier than
|
144
|
+
before. If you have a compound key and any of the keys have custom key generators, you need to pass in an array of
|
145
|
+
single-element hashes. This is in place to support Ruby 1.8, which does not have ordered hashes. Making sure the keys
|
146
|
+
are ordered is necessary to keep the compounds keys canonical (ie: deterministic).
|
147
|
+
|
148
|
+
### Finding data
|
149
|
+
|
150
|
+
Working on this...
|
data/Rakefile
ADDED
@@ -0,0 +1,26 @@
|
|
1
|
+
# -*- encoding: utf-8 -*-
|
2
|
+
$:.push File.expand_path("../lib", __FILE__)
|
3
|
+
require "active_column/version"
|
4
|
+
|
5
|
+
Gem::Specification.new do |s|
|
6
|
+
s.name = "active_column"
|
7
|
+
s.version = ActiveColumn::VERSION
|
8
|
+
s.platform = Gem::Platform::RUBY
|
9
|
+
s.authors = ["Michael Wynholds"]
|
10
|
+
s.email = ["mike@wynholds.com"]
|
11
|
+
s.homepage = "http://rubygems.org/gems/active_column"
|
12
|
+
s.summary = %q{Provides time line support for Cassandra}
|
13
|
+
s.description = %q{Provides time line support for Cassandra}
|
14
|
+
|
15
|
+
s.rubyforge_project = "active_column"
|
16
|
+
|
17
|
+
s.files = `git ls-files`.split("\n")
|
18
|
+
s.test_files = `git ls-files -- {test,spec,features}/*`.split("\n")
|
19
|
+
s.executables = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
|
20
|
+
s.require_paths = ["lib"]
|
21
|
+
|
22
|
+
s.add_dependency 'simple_uuid'
|
23
|
+
|
24
|
+
s.add_development_dependency 'cassandra'
|
25
|
+
s.add_development_dependency 'rspec'
|
26
|
+
end
|
@@ -0,0 +1,90 @@
|
|
1
|
+
module ActiveColumn
|
2
|
+
|
3
|
+
class Base
|
4
|
+
|
5
|
+
attr_reader :attributes
|
6
|
+
|
7
|
+
def initialize(attrs = {})
|
8
|
+
@attributes = attrs
|
9
|
+
end
|
10
|
+
|
11
|
+
def self.column_family(column_family = nil)
|
12
|
+
return @column_family if column_family.nil?
|
13
|
+
@column_family = column_family
|
14
|
+
end
|
15
|
+
|
16
|
+
def self.keys(*keys)
|
17
|
+
return @keys if keys.nil? || keys.empty?
|
18
|
+
flattened = ( keys.size == 1 && keys[0].is_a?(Array) ? keys[0] : keys )
|
19
|
+
@keys = flattened.collect { |k| KeyConfig.new(k) }
|
20
|
+
end
|
21
|
+
|
22
|
+
def save()
|
23
|
+
value = { SimpleUUID::UUID.new => self.to_json }
|
24
|
+
key_parts = self.class.keys.each_with_object( {} ) do |key_config, key_parts|
|
25
|
+
key_parts[key_config.key] = get_keys(key_config)
|
26
|
+
end
|
27
|
+
keys = self.class.generate_keys(key_parts)
|
28
|
+
|
29
|
+
keys.each do |key|
|
30
|
+
ActiveColumn.connection.insert(self.class.column_family, key, value)
|
31
|
+
end
|
32
|
+
|
33
|
+
self
|
34
|
+
end
|
35
|
+
|
36
|
+
def self.find(key_parts, options = {})
|
37
|
+
keys = generate_keys key_parts
|
38
|
+
ActiveColumn.connection.multi_get(column_family, keys, options).each_with_object( {} ) do |(user, row), results|
|
39
|
+
results[user] = row.to_a.collect { |(_uuid, col)| new(JSON.parse(col)) }
|
40
|
+
end
|
41
|
+
end
|
42
|
+
|
43
|
+
def to_json(*a)
|
44
|
+
@attributes.to_json(*a)
|
45
|
+
end
|
46
|
+
|
47
|
+
private
|
48
|
+
|
49
|
+
def get_keys(key_config)
|
50
|
+
key_config.func.nil? ? attributes[key_config.key] : self.send(key_config.func)
|
51
|
+
end
|
52
|
+
|
53
|
+
def self.generate_keys(key_parts)
|
54
|
+
if keys.size == 1
|
55
|
+
key_config = keys.first
|
56
|
+
value = key_parts.is_a?(Hash) ? key_parts[key_config.key] : key_parts
|
57
|
+
return value if value.is_a? Array
|
58
|
+
return [value]
|
59
|
+
end
|
60
|
+
|
61
|
+
values = keys.collect { |kc| key_parts[kc.key] }
|
62
|
+
product = values.reduce do |memo, key_part|
|
63
|
+
memo = [memo] unless memo.is_a? Array
|
64
|
+
key_part = [key_part] unless key_part.is_a? Array
|
65
|
+
memo.product key_part
|
66
|
+
end
|
67
|
+
|
68
|
+
product.collect { |p| p.join(':') }
|
69
|
+
end
|
70
|
+
|
71
|
+
end
|
72
|
+
|
73
|
+
class KeyConfig
|
74
|
+
attr_accessor :key, :func
|
75
|
+
|
76
|
+
def initialize(key_conf)
|
77
|
+
if key_conf.is_a?(Hash)
|
78
|
+
@key = key_conf.keys[0]
|
79
|
+
@func = key_conf[@key]
|
80
|
+
else
|
81
|
+
@key = key_conf
|
82
|
+
end
|
83
|
+
end
|
84
|
+
|
85
|
+
def to_s
|
86
|
+
"KeyConfig[#{key}, #{func or '-'}]"
|
87
|
+
end
|
88
|
+
end
|
89
|
+
|
90
|
+
end
|
@@ -0,0 +1,107 @@
|
|
1
|
+
require 'spec_helper'
|
2
|
+
|
3
|
+
describe ActiveColumn::Base do
|
4
|
+
|
5
|
+
describe '#save' do
|
6
|
+
|
7
|
+
context 'given a model with a single key' do
|
8
|
+
before do
|
9
|
+
@counter = Counter.new(:tweets, 'user1', 'user2', 'all')
|
10
|
+
end
|
11
|
+
|
12
|
+
context 'and an attribute key function' do
|
13
|
+
before do
|
14
|
+
Tweet.new( user_id: 'user1', message: 'just woke up' ).save
|
15
|
+
Tweet.new( user_id: 'user2', message: 'kinda hungry' ).save
|
16
|
+
end
|
17
|
+
|
18
|
+
it 'saves the model for the key' do
|
19
|
+
@counter.diff.should == [1, 1, 0]
|
20
|
+
end
|
21
|
+
end
|
22
|
+
|
23
|
+
context 'and a custom key function' do
|
24
|
+
before do
|
25
|
+
AggregatingTweet.new( user_id: 'user1', message: 'just woke up' ).save
|
26
|
+
AggregatingTweet.new( user_id: 'user2', message: 'kinda hungry' ).save
|
27
|
+
end
|
28
|
+
|
29
|
+
it 'saves the model for the keys' do
|
30
|
+
@counter.diff.should == [1, 1, 2]
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
34
|
+
|
35
|
+
context 'given a model with a compound key' do
|
36
|
+
before do
|
37
|
+
@counts = Counter.new(:tweet_dms, 'user1:friend1', 'user1:friend2', 'user1:all', 'all:friend1', 'all:friend2')
|
38
|
+
TweetDM.new( user_id: 'user1', recipient_ids: ['friend1', 'friend2'], message: 'feeling blue' ).save
|
39
|
+
TweetDM.new( user_id: 'user1', recipient_ids: ['friend2'], message: 'now im better' ).save
|
40
|
+
end
|
41
|
+
|
42
|
+
it 'saves the model for the combined compounds keys' do
|
43
|
+
@counts.diff.should == [1, 2, 2, 1, 2]
|
44
|
+
end
|
45
|
+
end
|
46
|
+
|
47
|
+
end
|
48
|
+
|
49
|
+
describe '.generate_keys' do
|
50
|
+
|
51
|
+
context 'given a simple key model' do
|
52
|
+
before do
|
53
|
+
@model = SimpleKey.new
|
54
|
+
end
|
55
|
+
|
56
|
+
context 'and a single key' do
|
57
|
+
it 'returns an array with the single key' do
|
58
|
+
keys = @model.class.send :generate_keys, '1'
|
59
|
+
keys.should == ['1']
|
60
|
+
end
|
61
|
+
end
|
62
|
+
|
63
|
+
context 'and an array of keys' do
|
64
|
+
it 'returns an array with the keys' do
|
65
|
+
keys = @model.class.send :generate_keys, ['1', '2', '3']
|
66
|
+
keys.should == ['1', '2', '3']
|
67
|
+
end
|
68
|
+
end
|
69
|
+
|
70
|
+
context 'and a map with a single key' do
|
71
|
+
it 'returns an array with the single key' do
|
72
|
+
keys = @model.class.send :generate_keys, { :one => '1' }
|
73
|
+
keys.should == ['1']
|
74
|
+
end
|
75
|
+
end
|
76
|
+
|
77
|
+
context 'and a map with an array with a single key' do
|
78
|
+
it 'returns an array with the single key' do
|
79
|
+
keys = @model.class.send :generate_keys, { :one => ['1'] }
|
80
|
+
keys.should == ['1']
|
81
|
+
end
|
82
|
+
end
|
83
|
+
end
|
84
|
+
|
85
|
+
context 'given a compound key model' do
|
86
|
+
before do
|
87
|
+
@model = CompoundKey.new
|
88
|
+
end
|
89
|
+
|
90
|
+
context 'and a map of keys' do
|
91
|
+
it 'returns an array of the keys put together' do
|
92
|
+
keys = @model.class.send :generate_keys, { :one => ['1', '2'], :two => ['a', 'b'], :three => 'Z' }
|
93
|
+
keys.should == ['1:a:Z', '1:b:Z', '2:a:Z', '2:b:Z']
|
94
|
+
end
|
95
|
+
end
|
96
|
+
|
97
|
+
context 'and a different map of keys' do
|
98
|
+
it 'returns an array of the keys put together' do
|
99
|
+
keys = @model.class.send :generate_keys, { :one => '1', :two => ['a', 'b'], :three => 'Z' }
|
100
|
+
keys.should == ['1:a:Z', '1:b:Z']
|
101
|
+
end
|
102
|
+
end
|
103
|
+
end
|
104
|
+
|
105
|
+
end
|
106
|
+
|
107
|
+
end
|
@@ -0,0 +1,65 @@
|
|
1
|
+
require 'spec_helper'
|
2
|
+
|
3
|
+
describe ActiveColumn::Base do
|
4
|
+
|
5
|
+
describe '.find' do
|
6
|
+
|
7
|
+
context 'given a model with a simple key' do
|
8
|
+
before do
|
9
|
+
Tweet.new( :user_id => 'user1', :message => 'Going running' ).save
|
10
|
+
Tweet.new( :user_id => 'user2', :message => 'Watching TV' ).save
|
11
|
+
Tweet.new( :user_id => 'user1', :message => 'Now im hungry' ).save
|
12
|
+
Tweet.new( :user_id => 'user1', :message => 'Now im full' ).save
|
13
|
+
end
|
14
|
+
|
15
|
+
context 'and finding some for a single key' do
|
16
|
+
before do
|
17
|
+
@found = Tweet.find( 'user1', :count => 3, :reversed => true )
|
18
|
+
end
|
19
|
+
|
20
|
+
it 'find all of the models' do
|
21
|
+
@found.size.should == 1
|
22
|
+
@found['user1'].size.should == 3
|
23
|
+
@found['user1'].collect { |t| t.attributes['message'] }.should == [ 'Now im full', 'Now im hungry', 'Going running' ]
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
context 'and finding some for multiple keys' do
|
28
|
+
before do
|
29
|
+
@found = Tweet.find( ['user1', 'user2'], :count => 1, :reversed => true )
|
30
|
+
end
|
31
|
+
|
32
|
+
it 'finds all of the models' do
|
33
|
+
@found.size.should == 2
|
34
|
+
@found['user1'].collect { |t| t.attributes['message'] }.should == [ 'Now im full' ]
|
35
|
+
@found['user2'].collect { |t| t.attributes['message'] }.should == [ 'Watching TV' ]
|
36
|
+
end
|
37
|
+
end
|
38
|
+
end
|
39
|
+
|
40
|
+
context 'given a model with a compound key' do
|
41
|
+
before do
|
42
|
+
TweetDM.new( :user_id => 'user1', :recipient_ids => [ 'friend1', 'friend2' ], :message => 'Need to do laundry' ).save
|
43
|
+
TweetDM.new( :user_id => 'user1', :recipient_ids => [ 'friend2', 'friend3' ], :message => 'My leg itches' ).save
|
44
|
+
end
|
45
|
+
|
46
|
+
context 'and finding some for both keys' do
|
47
|
+
before do
|
48
|
+
@found = TweetDM.find( { :user_id => ['user1', 'user2'], :recipient_id => ['friend1', 'friend2', 'all'] }, :count => 1, :reversed => true )
|
49
|
+
end
|
50
|
+
|
51
|
+
it 'finds all of the models' do
|
52
|
+
@found.size.should == 6
|
53
|
+
@found['user1:friend1'].collect { |t| t.attributes['message'] }.should == [ 'Need to do laundry' ]
|
54
|
+
@found['user1:friend2'].collect { |t| t.attributes['message'] }.should == [ 'My leg itches' ]
|
55
|
+
@found['user1:all'].collect { |t| t.attributes['message'] }.should == [ 'My leg itches' ]
|
56
|
+
@found['user2:friend1'].should == []
|
57
|
+
@found['user2:friend2'].should == []
|
58
|
+
@found['user2:all'].should == []
|
59
|
+
end
|
60
|
+
end
|
61
|
+
end
|
62
|
+
|
63
|
+
end
|
64
|
+
|
65
|
+
end
|
data/spec/spec_helper.rb
ADDED
@@ -0,0 +1,34 @@
|
|
1
|
+
require 'cassandra'
|
2
|
+
require 'active_column'
|
3
|
+
|
4
|
+
Dir[ File.expand_path("../support/**/*.rb", __FILE__) ].each {|f| require f}
|
5
|
+
|
6
|
+
$cassandra = ActiveColumn.connection = Cassandra.new('active_column', '127.0.0.1:9160')
|
7
|
+
$cassandra.clear_keyspace!
|
8
|
+
|
9
|
+
RSpec.configure do |config|
|
10
|
+
|
11
|
+
end
|
12
|
+
|
13
|
+
class Counter
|
14
|
+
def initialize(cf, *keys)
|
15
|
+
@cf = cf
|
16
|
+
@keys = keys
|
17
|
+
@counts = get_counts
|
18
|
+
end
|
19
|
+
|
20
|
+
def diff()
|
21
|
+
new_counts = get_counts
|
22
|
+
@keys.each_with_object( [] ) do |key, counts|
|
23
|
+
counts << new_counts[key] - @counts[key]
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
private
|
28
|
+
|
29
|
+
def get_counts
|
30
|
+
@keys.each_with_object( {} ) do |key, counts|
|
31
|
+
counts[key] = $cassandra.count_columns(@cf, key)
|
32
|
+
end
|
33
|
+
end
|
34
|
+
end
|
@@ -0,0 +1,54 @@
|
|
1
|
+
<Storage>
|
2
|
+
<ClusterName>ActiveColumn</ClusterName>
|
3
|
+
<AutoBootstrap>false</AutoBootstrap>
|
4
|
+
<HintedHandoffEnabled>true</HintedHandoffEnabled>
|
5
|
+
|
6
|
+
<Keyspaces>
|
7
|
+
<Keyspace Name="active_column">
|
8
|
+
<ColumnFamily Name="time" CompareWith="TimeUUIDType"/>
|
9
|
+
<ColumnFamily Name="tweets" CompareWith="TimeUUIDType"/>
|
10
|
+
<ColumnFamily Name="tweet_dms" CompareWith="TimeUUIDType"/>
|
11
|
+
<ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
|
12
|
+
<ReplicationFactor>1</ReplicationFactor>
|
13
|
+
<EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
|
14
|
+
</Keyspace>
|
15
|
+
</Keyspaces>
|
16
|
+
|
17
|
+
<Authenticator>org.apache.cassandra.auth.AllowAllAuthenticator</Authenticator>
|
18
|
+
<Partitioner>org.apache.cassandra.dht.RandomPartitioner</Partitioner>
|
19
|
+
<InitialToken></InitialToken>
|
20
|
+
|
21
|
+
<SavedCachesDirectory>/var/lib/cassandra/saved_caches</SavedCachesDirectory>
|
22
|
+
<CommitLogDirectory>/var/lib/cassandra/commitlog</CommitLogDirectory>
|
23
|
+
<DataFileDirectories>
|
24
|
+
<DataFileDirectory>/var/lib/cassandra/data</DataFileDirectory>
|
25
|
+
</DataFileDirectories>
|
26
|
+
|
27
|
+
<Seeds>
|
28
|
+
<Seed>127.0.0.1</Seed>
|
29
|
+
</Seeds>
|
30
|
+
|
31
|
+
<RpcTimeoutInMillis>10000</RpcTimeoutInMillis>
|
32
|
+
<CommitLogRotationThresholdInMB>128</CommitLogRotationThresholdInMB>
|
33
|
+
<ListenAddress>localhost</ListenAddress>
|
34
|
+
<StoragePort>7000</StoragePort>
|
35
|
+
<ThriftAddress>localhost</ThriftAddress>
|
36
|
+
<ThriftPort>9160</ThriftPort>
|
37
|
+
<ThriftFramedTransport>false</ThriftFramedTransport>
|
38
|
+
|
39
|
+
<DiskAccessMode>auto</DiskAccessMode>
|
40
|
+
<RowWarningThresholdInMB>512</RowWarningThresholdInMB>
|
41
|
+
<SlicedBufferSizeInKB>64</SlicedBufferSizeInKB>
|
42
|
+
<FlushDataBufferSizeInMB>32</FlushDataBufferSizeInMB>
|
43
|
+
<FlushIndexBufferSizeInMB>8</FlushIndexBufferSizeInMB>
|
44
|
+
<ColumnIndexSizeInKB>64</ColumnIndexSizeInKB>
|
45
|
+
<MemtableThroughputInMB>64</MemtableThroughputInMB>
|
46
|
+
<BinaryMemtableThroughputInMB>256</BinaryMemtableThroughputInMB>
|
47
|
+
<MemtableOperationsInMillions>0.3</MemtableOperationsInMillions>
|
48
|
+
<MemtableFlushAfterMinutes>60</MemtableFlushAfterMinutes>
|
49
|
+
<ConcurrentReads>8</ConcurrentReads>
|
50
|
+
<ConcurrentWrites>32</ConcurrentWrites>
|
51
|
+
<CommitLogSync>periodic</CommitLogSync>
|
52
|
+
<CommitLogSyncPeriodInMS>10000</CommitLogSyncPeriodInMS>
|
53
|
+
<GCGraceSeconds>864000</GCGraceSeconds>
|
54
|
+
</Storage>
|
@@ -0,0 +1,14 @@
|
|
1
|
+
class TweetDM < ActiveColumn::Base
|
2
|
+
|
3
|
+
column_family :tweet_dms
|
4
|
+
keys [ { :user_id => :user_keys }, { :recipient_id => :recipient_keys } ]
|
5
|
+
|
6
|
+
def user_keys
|
7
|
+
[ attributes[:user_id], 'all' ]
|
8
|
+
end
|
9
|
+
|
10
|
+
def recipient_keys
|
11
|
+
attributes[:recipient_ids] + ['all']
|
12
|
+
end
|
13
|
+
|
14
|
+
end
|
metadata
ADDED
@@ -0,0 +1,131 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: active_column
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
prerelease: false
|
5
|
+
segments:
|
6
|
+
- 0
|
7
|
+
- 0
|
8
|
+
- 1
|
9
|
+
version: 0.0.1
|
10
|
+
platform: ruby
|
11
|
+
authors:
|
12
|
+
- Michael Wynholds
|
13
|
+
autorequire:
|
14
|
+
bindir: bin
|
15
|
+
cert_chain: []
|
16
|
+
|
17
|
+
date: 2010-12-12 00:00:00 -08:00
|
18
|
+
default_executable:
|
19
|
+
dependencies:
|
20
|
+
- !ruby/object:Gem::Dependency
|
21
|
+
name: simple_uuid
|
22
|
+
prerelease: false
|
23
|
+
requirement: &id001 !ruby/object:Gem::Requirement
|
24
|
+
none: false
|
25
|
+
requirements:
|
26
|
+
- - ">="
|
27
|
+
- !ruby/object:Gem::Version
|
28
|
+
segments:
|
29
|
+
- 0
|
30
|
+
version: "0"
|
31
|
+
type: :runtime
|
32
|
+
version_requirements: *id001
|
33
|
+
- !ruby/object:Gem::Dependency
|
34
|
+
name: cassandra
|
35
|
+
prerelease: false
|
36
|
+
requirement: &id002 !ruby/object:Gem::Requirement
|
37
|
+
none: false
|
38
|
+
requirements:
|
39
|
+
- - ">="
|
40
|
+
- !ruby/object:Gem::Version
|
41
|
+
segments:
|
42
|
+
- 0
|
43
|
+
version: "0"
|
44
|
+
type: :development
|
45
|
+
version_requirements: *id002
|
46
|
+
- !ruby/object:Gem::Dependency
|
47
|
+
name: rspec
|
48
|
+
prerelease: false
|
49
|
+
requirement: &id003 !ruby/object:Gem::Requirement
|
50
|
+
none: false
|
51
|
+
requirements:
|
52
|
+
- - ">="
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
segments:
|
55
|
+
- 0
|
56
|
+
version: "0"
|
57
|
+
type: :development
|
58
|
+
version_requirements: *id003
|
59
|
+
description: Provides time line support for Cassandra
|
60
|
+
email:
|
61
|
+
- mike@wynholds.com
|
62
|
+
executables: []
|
63
|
+
|
64
|
+
extensions: []
|
65
|
+
|
66
|
+
extra_rdoc_files: []
|
67
|
+
|
68
|
+
files:
|
69
|
+
- .gitignore
|
70
|
+
- .rvmrc
|
71
|
+
- Gemfile
|
72
|
+
- Gemfile.lock
|
73
|
+
- README.html
|
74
|
+
- README.md
|
75
|
+
- Rakefile
|
76
|
+
- active_column.gemspec
|
77
|
+
- lib/active_column.rb
|
78
|
+
- lib/active_column/base.rb
|
79
|
+
- lib/active_column/connection.rb
|
80
|
+
- lib/active_column/version.rb
|
81
|
+
- spec/active_column/base_crud_spec.rb
|
82
|
+
- spec/active_column/base_finders_spec.rb
|
83
|
+
- spec/spec_helper.rb
|
84
|
+
- spec/support/aggregating_tweet.rb
|
85
|
+
- spec/support/compound_key.rb
|
86
|
+
- spec/support/config/storage-conf.xml
|
87
|
+
- spec/support/simple_key.rb
|
88
|
+
- spec/support/tweet.rb
|
89
|
+
- spec/support/tweet_dm.rb
|
90
|
+
has_rdoc: true
|
91
|
+
homepage: http://rubygems.org/gems/active_column
|
92
|
+
licenses: []
|
93
|
+
|
94
|
+
post_install_message:
|
95
|
+
rdoc_options: []
|
96
|
+
|
97
|
+
require_paths:
|
98
|
+
- lib
|
99
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
100
|
+
none: false
|
101
|
+
requirements:
|
102
|
+
- - ">="
|
103
|
+
- !ruby/object:Gem::Version
|
104
|
+
segments:
|
105
|
+
- 0
|
106
|
+
version: "0"
|
107
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
108
|
+
none: false
|
109
|
+
requirements:
|
110
|
+
- - ">="
|
111
|
+
- !ruby/object:Gem::Version
|
112
|
+
segments:
|
113
|
+
- 0
|
114
|
+
version: "0"
|
115
|
+
requirements: []
|
116
|
+
|
117
|
+
rubyforge_project: active_column
|
118
|
+
rubygems_version: 1.3.7
|
119
|
+
signing_key:
|
120
|
+
specification_version: 3
|
121
|
+
summary: Provides time line support for Cassandra
|
122
|
+
test_files:
|
123
|
+
- spec/active_column/base_crud_spec.rb
|
124
|
+
- spec/active_column/base_finders_spec.rb
|
125
|
+
- spec/spec_helper.rb
|
126
|
+
- spec/support/aggregating_tweet.rb
|
127
|
+
- spec/support/compound_key.rb
|
128
|
+
- spec/support/config/storage-conf.xml
|
129
|
+
- spec/support/simple_key.rb
|
130
|
+
- spec/support/tweet.rb
|
131
|
+
- spec/support/tweet_dm.rb
|