related 0.4.0 → 0.5.0
Sign up to get free protection for your applications and to get access to all the features.
- data/CHANGELOG +5 -0
- data/README.md +64 -0
- data/lib/related/data_flow.rb +55 -0
- data/lib/related/entity.rb +1 -1
- data/lib/related/follower.rb +2 -5
- data/lib/related/node.rb +20 -0
- data/lib/related/relationship.rb +9 -2
- data/lib/related/version.rb +1 -1
- data/lib/related.rb +2 -0
- data/test/data_flow_test.rb +55 -0
- data/test/model_test.rb +4 -4
- data/test/related_test.rb +17 -0
- metadata +5 -9
data/CHANGELOG
CHANGED
data/README.md
CHANGED
@@ -72,6 +72,7 @@ node.outgoing(:friends).relationships
|
|
72
72
|
node.outgoing(:friends).nodes
|
73
73
|
node.outgoing(:friends).limit(5)
|
74
74
|
node.outgoing(:friends).options(:fields => ..., :model => ...)
|
75
|
+
node1.outgoing(:friends).relationships.find(node2)
|
75
76
|
node1.path_to(node2).outgoing(:friends).depth(3)
|
76
77
|
node1.shortest_path_to(node2).outgoing(:friends).depth(3)
|
77
78
|
```
|
@@ -268,6 +269,69 @@ user1.following_count
|
|
268
269
|
The two nodes does not need to be of the same type. You can for example have
|
269
270
|
a User following a Page or whatever makes sense in your app.
|
270
271
|
|
272
|
+
Real-time Stream Processing
|
273
|
+
---------------------------
|
274
|
+
|
275
|
+
When working with graphs you often want to take the rich and interconnected
|
276
|
+
web of data and actually do something with it. Stream processing is a powerful
|
277
|
+
and flexible way to do that. It allows you to implement complex graph
|
278
|
+
algorithms in a scalable way that is also easy to understand and work with.
|
279
|
+
|
280
|
+
Stream processing in Related works by defining a data flow that new or
|
281
|
+
existing data will be streamed through. A data flow is triggered when a
|
282
|
+
Relationship is created, updated or deleted. You setup data flows for
|
283
|
+
different relationship types, so for example when a "friend" relationship
|
284
|
+
between two nodes is created or updated that relationship will be
|
285
|
+
automatically sent through the data flows you have defined for the "friend"
|
286
|
+
type.
|
287
|
+
|
288
|
+
A data flow can consist of one or more steps and can branch out in a tree.
|
289
|
+
You define the steps for a data flow using a simple Hash syntax.
|
290
|
+
|
291
|
+
```ruby
|
292
|
+
Related.data_flow :comment, Tokenize => { CountWords => { TotalSum => nil, MovingAverage => nil } }
|
293
|
+
```
|
294
|
+
|
295
|
+
In the example above a new comment will first sent to the Tokenize step that
|
296
|
+
will split the comment text into words. The list of words will then
|
297
|
+
automatically be sent to the CountWords step that will count the number of
|
298
|
+
unique words. That number will then be sent to both the TotalSum step that
|
299
|
+
adds the number to a global counter as well as the MovingAverage step that
|
300
|
+
will calculate and store a moving average. The nil indicates the end of the
|
301
|
+
data flow. You can define as many data flows for a relationship type as you
|
302
|
+
want.
|
303
|
+
|
304
|
+
A data flow step is simply a Ruby class that responds to the `process` message
|
305
|
+
and takes a single argument that holds the input data. Any data yielded from
|
306
|
+
the process method will be automatically sent to the next step in the data
|
307
|
+
flow. The only limitation is that the data sent between steps is a Hash and
|
308
|
+
only contains JSON serializable data. The first step in the data flow will
|
309
|
+
receive the Relationship object that triggered it as a Ruby hash with all of
|
310
|
+
its attributes.
|
311
|
+
|
312
|
+
```ruby
|
313
|
+
class Tokenize
|
314
|
+
def self.process(data)
|
315
|
+
data['text'].split(' ').each do |word|
|
316
|
+
yield({ :word => word })
|
317
|
+
end
|
318
|
+
end
|
319
|
+
end
|
320
|
+
```
|
321
|
+
|
322
|
+
To actually run the data flows you have defined you need to start one or more
|
323
|
+
data flow workers. Related uses Resque which supplies persistent queues and
|
324
|
+
reliable workers. If you don't have Resque required in your application
|
325
|
+
Related will simply run the work flow directly in process instead which can
|
326
|
+
be useful when testing, but is not recommended for production.
|
327
|
+
|
328
|
+
To start a stream processor:
|
329
|
+
|
330
|
+
$ QUEUE=related rake resque:work
|
331
|
+
|
332
|
+
You can start as many stream processors as you may need to scale
|
333
|
+
up.
|
334
|
+
|
271
335
|
Development
|
272
336
|
-----------
|
273
337
|
|
@@ -0,0 +1,55 @@
|
|
1
|
+
module Related
|
2
|
+
module DataFlow
|
3
|
+
|
4
|
+
def data_flow(name, steps)
|
5
|
+
@data_flows ||= {}
|
6
|
+
@data_flows[name.to_sym] ||= []
|
7
|
+
@data_flows[name.to_sym] << steps
|
8
|
+
end
|
9
|
+
|
10
|
+
def data_flows
|
11
|
+
@data_flows
|
12
|
+
end
|
13
|
+
|
14
|
+
def clear_data_flows
|
15
|
+
@data_flows = {}
|
16
|
+
end
|
17
|
+
|
18
|
+
def execute_data_flow(name_or_flow, data)
|
19
|
+
@data_flows ||= {}
|
20
|
+
if name_or_flow.is_a?(Hash)
|
21
|
+
enqueue_flow(name_or_flow, data)
|
22
|
+
else
|
23
|
+
flows = @data_flows[name_or_flow.to_sym] || []
|
24
|
+
flows.each do |flow|
|
25
|
+
enqueue_flow(flow, data)
|
26
|
+
end
|
27
|
+
end
|
28
|
+
end
|
29
|
+
|
30
|
+
class DataFlowJob
|
31
|
+
@queue = :related
|
32
|
+
def self.perform(flow, data)
|
33
|
+
flow.keys.each do |key|
|
34
|
+
step = key.constantize
|
35
|
+
step.perform(data) do |result|
|
36
|
+
if flow[key]
|
37
|
+
Related.execute_data_flow(flow[key], result)
|
38
|
+
end
|
39
|
+
end
|
40
|
+
end
|
41
|
+
end
|
42
|
+
end
|
43
|
+
|
44
|
+
protected
|
45
|
+
|
46
|
+
def enqueue_flow(flow, data)
|
47
|
+
if defined?(Resque)
|
48
|
+
Resque.enqueue(DataFlowJob, flow, data)
|
49
|
+
else
|
50
|
+
DataFlowJob.perform(JSON.parse(flow.to_json), JSON.parse(data.to_json))
|
51
|
+
end
|
52
|
+
end
|
53
|
+
|
54
|
+
end
|
55
|
+
end
|
data/lib/related/entity.rb
CHANGED
@@ -102,7 +102,7 @@ module Related
|
|
102
102
|
|
103
103
|
def self.find(*args)
|
104
104
|
options = args.size > 1 && args.last.is_a?(Hash) ? args.pop : {}
|
105
|
-
args.size == 1 && args.first.is_a?(
|
105
|
+
args.size == 1 && !args.first.is_a?(Array) ?
|
106
106
|
find_one(args.first, options) :
|
107
107
|
find_many(args.flatten, options)
|
108
108
|
end
|
data/lib/related/follower.rb
CHANGED
data/lib/related/node.rb
CHANGED
@@ -144,6 +144,18 @@ module Related
|
|
144
144
|
end
|
145
145
|
end
|
146
146
|
|
147
|
+
def find(node)
|
148
|
+
if @result_type == :nodes
|
149
|
+
if Related.redis.sismember(key, node.to_s)
|
150
|
+
Related::Node.find(node.to_s, @options)
|
151
|
+
end
|
152
|
+
else
|
153
|
+
if id = Related.redis.get(dir_key(node))
|
154
|
+
Related::Relationship.find(id, @options)
|
155
|
+
end
|
156
|
+
end
|
157
|
+
end
|
158
|
+
|
147
159
|
def union(query)
|
148
160
|
@result_type = :nodes
|
149
161
|
@result = Related.redis.sunion(key, query.key)
|
@@ -193,6 +205,14 @@ module Related
|
|
193
205
|
end
|
194
206
|
end
|
195
207
|
|
208
|
+
def dir_key(node)
|
209
|
+
if @direction == :out
|
210
|
+
"#{@node.to_s}:#{@relationship_type}:#{node.to_s}"
|
211
|
+
elsif @direction == :in
|
212
|
+
"#{node.to_s}:#{@relationship_type}:#{@node.to_s}"
|
213
|
+
end
|
214
|
+
end
|
215
|
+
|
196
216
|
def query
|
197
217
|
self
|
198
218
|
end
|
data/lib/related/relationship.rb
CHANGED
@@ -56,6 +56,10 @@ module Related
|
|
56
56
|
end
|
57
57
|
end
|
58
58
|
|
59
|
+
def dir_key
|
60
|
+
"#{self.start_node_id}:#{self.label}:#{self.end_node_id}"
|
61
|
+
end
|
62
|
+
|
59
63
|
def self.weight_for(relationship, direction)
|
60
64
|
if @weight
|
61
65
|
relationship.instance_exec(direction, &@weight).to_i
|
@@ -71,7 +75,9 @@ module Related
|
|
71
75
|
Related.redis.zadd(r_key(:in), self.class.weight_for(self, :in), self.id)
|
72
76
|
Related.redis.sadd(n_key(:out), self.end_node_id)
|
73
77
|
Related.redis.sadd(n_key(:in), self.start_node_id)
|
78
|
+
Related.redis.set(dir_key, self.id)
|
74
79
|
end
|
80
|
+
Related.execute_data_flow(self.label, self)
|
75
81
|
self
|
76
82
|
end
|
77
83
|
|
@@ -80,9 +86,8 @@ module Related
|
|
80
86
|
super
|
81
87
|
Related.redis.zadd(r_key(:out), self.class.weight_for(self, :out), self.id)
|
82
88
|
Related.redis.zadd(r_key(:in), self.class.weight_for(self, :in), self.id)
|
83
|
-
Related.redis.sadd(n_key(:out), self.end_node_id)
|
84
|
-
Related.redis.sadd(n_key(:in), self.start_node_id)
|
85
89
|
end
|
90
|
+
Related.execute_data_flow(self.label, self)
|
86
91
|
self
|
87
92
|
end
|
88
93
|
|
@@ -92,8 +97,10 @@ module Related
|
|
92
97
|
Related.redis.zrem(r_key(:in), self.id)
|
93
98
|
Related.redis.srem(n_key(:out), self.end_node_id)
|
94
99
|
Related.redis.srem(n_key(:in), self.start_node_id)
|
100
|
+
Related.redis.del(dir_key)
|
95
101
|
super
|
96
102
|
end
|
103
|
+
Related.execute_data_flow(self.label, self)
|
97
104
|
self
|
98
105
|
end
|
99
106
|
|
data/lib/related/version.rb
CHANGED
data/lib/related.rb
CHANGED
@@ -0,0 +1,55 @@
|
|
1
|
+
require File.expand_path('test/test_helper')
|
2
|
+
|
3
|
+
class ModelTest < ActiveModel::TestCase
|
4
|
+
|
5
|
+
class StepOne
|
6
|
+
def self.perform(data)
|
7
|
+
yield({ :text => data['text'] })
|
8
|
+
end
|
9
|
+
end
|
10
|
+
|
11
|
+
class StepTwo
|
12
|
+
def self.perform(data)
|
13
|
+
yield({ :id => 'StepTwo', :text => data['text'].downcase })
|
14
|
+
end
|
15
|
+
end
|
16
|
+
|
17
|
+
class StepThree
|
18
|
+
def self.perform(data)
|
19
|
+
yield({ :id => 'StepThree', :text => data['text'].upcase })
|
20
|
+
end
|
21
|
+
end
|
22
|
+
|
23
|
+
class LastStep
|
24
|
+
def self.perform(data)
|
25
|
+
Related.redis.set("DataFlowResult#{data['id']}", data['text'])
|
26
|
+
end
|
27
|
+
end
|
28
|
+
|
29
|
+
def setup
|
30
|
+
Related.redis.flushall
|
31
|
+
end
|
32
|
+
|
33
|
+
def teardown
|
34
|
+
Related.clear_data_flows
|
35
|
+
end
|
36
|
+
|
37
|
+
def test_defining_a_data_flow
|
38
|
+
Related.data_flow :like, StepOne => { StepTwo => nil }
|
39
|
+
assert_equal({ :like => [{ StepOne => { StepTwo => nil } }] }, Related.data_flows)
|
40
|
+
end
|
41
|
+
|
42
|
+
def test_executing_a_simple_data_flow
|
43
|
+
Related.data_flow :like, StepOne => { LastStep => nil }
|
44
|
+
Related::Relationship.create(:like, Related::Node.create, Related::Node.create, :text => 'Hello world!')
|
45
|
+
assert_equal 'Hello world!', Related.redis.get('DataFlowResult')
|
46
|
+
end
|
47
|
+
|
48
|
+
def test_executing_a_complicated_data_flow
|
49
|
+
Related.data_flow :like, StepOne => { StepTwo => { LastStep => nil }, StepThree => { LastStep => nil } }
|
50
|
+
Related::Relationship.create(:like, Related::Node.create, Related::Node.create, :text => 'Hello world!')
|
51
|
+
assert_equal 'hello world!', Related.redis.get('DataFlowResultStepTwo')
|
52
|
+
assert_equal 'HELLO WORLD!', Related.redis.get('DataFlowResultStepThree')
|
53
|
+
end
|
54
|
+
|
55
|
+
end
|
data/test/model_test.rb
CHANGED
@@ -72,10 +72,10 @@ class ModelTest < ActiveModel::TestCase
|
|
72
72
|
rel2 = Like.create(:like, node1, node3, :in_score => 2, :out_score => 2)
|
73
73
|
rel3 = Like.create(:like, node2, node1, :in_score => 1, :out_score => 1)
|
74
74
|
rel4 = Like.create(:like, node3, node1, :in_score => 2, :out_score => 2)
|
75
|
-
assert_equal [rel2,rel1], node1.outgoing(:like).relationships.options(:model => lambda { Like }).to_a
|
76
|
-
assert_equal [rel4,rel3], node1.incoming(:like).relationships.options(:model => lambda { Like }).to_a
|
77
|
-
assert_equal [rel1], node1.outgoing(:like).relationships.options(:model => lambda { Like }).per_page(2).page(rel2).to_a
|
78
|
-
assert_equal [rel3], node1.incoming(:like).relationships.options(:model => lambda { Like }).per_page(2).page(rel4).to_a
|
75
|
+
assert_equal [rel2,rel1], node1.outgoing(:like).relationships.options(:model => lambda {|a| Like }).to_a
|
76
|
+
assert_equal [rel4,rel3], node1.incoming(:like).relationships.options(:model => lambda {|a| Like }).to_a
|
77
|
+
assert_equal [rel1], node1.outgoing(:like).relationships.options(:model => lambda {|a| Like }).per_page(2).page(rel2).to_a
|
78
|
+
assert_equal [rel3], node1.incoming(:like).relationships.options(:model => lambda {|a| Like }).per_page(2).page(rel4).to_a
|
79
79
|
end
|
80
80
|
|
81
81
|
end
|
data/test/related_test.rb
CHANGED
@@ -45,6 +45,9 @@ class RelatedTest < Test::Unit::TestCase
|
|
45
45
|
assert_raises Related::NotFound do
|
46
46
|
Related::Node.find('foo')
|
47
47
|
end
|
48
|
+
assert_raises Related::NotFound do
|
49
|
+
Related::Node.find(nil)
|
50
|
+
end
|
48
51
|
end
|
49
52
|
|
50
53
|
def test_can_update_node_with_new_attributes
|
@@ -295,4 +298,18 @@ class RelatedTest < Test::Unit::TestCase
|
|
295
298
|
Related.root.save
|
296
299
|
end
|
297
300
|
|
301
|
+
def test_find
|
302
|
+
node1 = Related::Node.create
|
303
|
+
node2 = Related::Node.create
|
304
|
+
rel = Related::Relationship.create(:friend, node1, node2)
|
305
|
+
assert_equal node2, node1.outgoing(:friend).find(node2)
|
306
|
+
assert_equal node1, node2.incoming(:friend).find(node1)
|
307
|
+
assert_equal nil, node1.outgoing(:friend).find(node1)
|
308
|
+
assert_equal nil, node2.incoming(:friend).find(node2)
|
309
|
+
assert_equal rel, node1.outgoing(:friend).relationships.find(node2)
|
310
|
+
assert_equal rel, node2.incoming(:friend).relationships.find(node1)
|
311
|
+
assert_equal nil, node1.outgoing(:friend).relationships.find(node1)
|
312
|
+
assert_equal nil, node2.incoming(:friend).relationships.find(node2)
|
313
|
+
end
|
314
|
+
|
298
315
|
end
|
metadata
CHANGED
@@ -1,13 +1,12 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: related
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
hash: 15
|
5
4
|
prerelease: false
|
6
5
|
segments:
|
7
6
|
- 0
|
8
|
-
-
|
7
|
+
- 5
|
9
8
|
- 0
|
10
|
-
version: 0.
|
9
|
+
version: 0.5.0
|
11
10
|
platform: ruby
|
12
11
|
authors:
|
13
12
|
- Niklas Holmgren
|
@@ -15,7 +14,7 @@ autorequire:
|
|
15
14
|
bindir: bin
|
16
15
|
cert_chain: []
|
17
16
|
|
18
|
-
date: 2011-
|
17
|
+
date: 2011-11-02 00:00:00 +01:00
|
19
18
|
default_executable:
|
20
19
|
dependencies:
|
21
20
|
- !ruby/object:Gem::Dependency
|
@@ -26,7 +25,6 @@ dependencies:
|
|
26
25
|
requirements:
|
27
26
|
- - ">"
|
28
27
|
- !ruby/object:Gem::Version
|
29
|
-
hash: 15
|
30
28
|
segments:
|
31
29
|
- 2
|
32
30
|
- 0
|
@@ -42,7 +40,6 @@ dependencies:
|
|
42
40
|
requirements:
|
43
41
|
- - ">"
|
44
42
|
- !ruby/object:Gem::Version
|
45
|
-
hash: 63
|
46
43
|
segments:
|
47
44
|
- 0
|
48
45
|
- 8
|
@@ -58,7 +55,6 @@ dependencies:
|
|
58
55
|
requirements:
|
59
56
|
- - ">="
|
60
57
|
- !ruby/object:Gem::Version
|
61
|
-
hash: 3
|
62
58
|
segments:
|
63
59
|
- 0
|
64
60
|
version: "0"
|
@@ -78,6 +74,7 @@ files:
|
|
78
74
|
- Rakefile
|
79
75
|
- LICENSE
|
80
76
|
- CHANGELOG
|
77
|
+
- lib/related/data_flow.rb
|
81
78
|
- lib/related/entity.rb
|
82
79
|
- lib/related/exceptions.rb
|
83
80
|
- lib/related/follower.rb
|
@@ -88,6 +85,7 @@ files:
|
|
88
85
|
- lib/related/version.rb
|
89
86
|
- lib/related.rb
|
90
87
|
- test/active_model_test.rb
|
88
|
+
- test/data_flow_test.rb
|
91
89
|
- test/follower_test.rb
|
92
90
|
- test/model_test.rb
|
93
91
|
- test/performance_test.rb
|
@@ -108,7 +106,6 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
108
106
|
requirements:
|
109
107
|
- - ">="
|
110
108
|
- !ruby/object:Gem::Version
|
111
|
-
hash: 3
|
112
109
|
segments:
|
113
110
|
- 0
|
114
111
|
version: "0"
|
@@ -117,7 +114,6 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
117
114
|
requirements:
|
118
115
|
- - ">="
|
119
116
|
- !ruby/object:Gem::Version
|
120
|
-
hash: 3
|
121
117
|
segments:
|
122
118
|
- 0
|
123
119
|
version: "0"
|