monsoon-droplet 1.0.0 → 1.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +29 -24
- data/lib/monsoon.rb +9 -0
- data/lib/monsoon/droplet.rb +9 -7
- data/lib/monsoon/streams/console.rb +1 -1
- data/lib/monsoon/streams/kinesis.rb +2 -2
- data/lib/monsoon/version.rb +1 -1
- data/lib/monsoon/versions_schema.rb +15 -6
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 71c6d61b46942b8e4b68c0e52742786e9d6d5809
|
4
|
+
data.tar.gz: 220a3aafa400bd53007a7472008eaf4051d9b951
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: c4d7e57af9b1e1f02e71ea69fe87003fe4bb90f53303709c18fbbc8be45ab59572c13cd272ad1f03e38e5b0188ad8922e56e4d05dcea867ac0302171a7cfb945
|
7
|
+
data.tar.gz: f61e8e456527c79551542091ccf7f94dbc75b68ca6abe2f098c6ff773b084c864e361fff65541ef22bdc9d045b0c2d70f0747ca27f3ab7f00743ad09c88177a7
|
data/README.md
CHANGED
@@ -32,17 +32,18 @@ require 'monsoon/streams/kinesis'
|
|
32
32
|
require 'monsoon/streams/console'
|
33
33
|
|
34
34
|
Monsoon.versions_schema = {
|
35
|
-
"
|
36
|
-
"
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
|
35
|
+
"transcodes" => {
|
36
|
+
"download_complete" => {
|
37
|
+
"1.0" => ["filename", "media_id"],
|
38
|
+
"2.0" => ["url", "media_id"]
|
39
|
+
},
|
40
|
+
"transcode_complete" => {
|
41
|
+
"1.1" => ["filename", "media_id"],
|
42
|
+
"2.0" => ["media_id", "status"]
|
43
|
+
}
|
44
44
|
}
|
45
45
|
}
|
46
|
+
Monsoon.default_stream = 'transcodes'
|
46
47
|
|
47
48
|
# if using the Kinesis stream...
|
48
49
|
ENV['AWS_ACCESS_KEY_ID'] = "YOUR AWS ACCESS KEY"
|
@@ -60,39 +61,43 @@ require 'monsoon/streams/kinesis'
|
|
60
61
|
|
61
62
|
This will add the built-in AWS Kinesis adapter. `monsoon/streams/console` is also built-in for testing by writing your streams to STDOUT.
|
62
63
|
|
63
|
-
You can write your own stream adapter by creating a class that implements `#put_records(stream_name, records_array)` and then add it to Monsoon with `Monsoon.streams << YourNewAdapter.new`. See `Monsoon::Streams::Console` for an example.
|
64
|
+
You can write your own stream adapter by creating a class that implements `#put_records(stream_name, records_array, options_hsh = {})` and then add it to Monsoon with `Monsoon.streams << YourNewAdapter.new`. See `Monsoon::Streams::Console` for an example.
|
64
65
|
|
65
66
|
Monsoon will send your messages to all added streams by default.
|
66
67
|
|
68
|
+
Note on Kinesis Stream: Kinesis requires a partition key when streaming a message. Monsoon will look for a `partition_key` in your droplet's `options` hash. If no `partition_key` is provided, Monsoon will default to partitioning on the event name or droplet version. If neither of those are available, Monsoon will send everything to the `monsoon` partition.
|
69
|
+
|
67
70
|
### Versions Schema
|
68
71
|
|
69
|
-
To auto-generate versioned copies of your message, configure Monsoon with your versions schema. This is just a hash of stream names and keys that tells Monsoon, given a stream name, here's a list of versions we support and the keys each version should include in its message. For example, if you have the following schema defined:
|
72
|
+
To auto-generate versioned copies of your message, configure Monsoon with your versions schema. This is just a hash of stream names and keys that tells Monsoon, given a stream name and event name, here's a list of versions we support and the keys each version should include in its message. For example, if you have the following schema defined:
|
70
73
|
|
71
74
|
```ruby
|
72
75
|
Monsoon.versions_schema = {
|
73
|
-
"
|
74
|
-
"
|
75
|
-
|
76
|
-
|
77
|
-
|
78
|
-
|
79
|
-
|
80
|
-
|
81
|
-
|
76
|
+
"transcodes" => {
|
77
|
+
"download_complete" => {
|
78
|
+
"1.0" => ["filename", "media_id"],
|
79
|
+
"2.0" => ["url", "media_id"]
|
80
|
+
},
|
81
|
+
"transcode_complete" => {
|
82
|
+
"1.1" => ["filename", "media_id"],
|
83
|
+
"2.0" => ["media_id", "status"]
|
84
|
+
}
|
82
85
|
}
|
83
86
|
}
|
84
87
|
```
|
85
88
|
|
86
|
-
and you want to send the message `{
|
89
|
+
and you want to send the message `{stream: 'transcodes', event: 'download_complete', filename: 'my_movie.mp4', media_id: 5, url: 'example.com/my_movie.mp4'}`, Monsoon will automatically generate 2 versions of your message for each defined version (1.0 and 2.0). The versioned message will only include the keys needed for that version, as well as 2 new keys: 'droplet_version' and 'droplet_deprecated'. `droplet_version` tells the receiving client which version they're reading, and `version_deprecated` is true if there is a newer version defined in the schema.
|
90
|
+
|
91
|
+
If the original message does not include all of the keys needed for an older version (e.g., `filename` is left out in the above example), then Monsoon will only send the versions it has the data for (e.g., it will only send v2.0).
|
87
92
|
|
88
|
-
|
93
|
+
Since many apps will only use 1 stream, you can also set `Monsoon.default_stream` to avoid passing the `stream` key with each message.
|
89
94
|
|
90
95
|
### Droplets
|
91
96
|
|
92
97
|
To send a message after you've configured Monsoon, just create a new Droplet and stream it:
|
93
98
|
|
94
99
|
```ruby
|
95
|
-
droplet = Monsoon::Droplet.new(
|
100
|
+
droplet = Monsoon::Droplet.new({stream: 'transcodes', event: 'download_complete', filename: 'my_movie.mp4', media_id: 5, url: 'example.com/my_movie.mp4'})
|
96
101
|
droplet.stream
|
97
102
|
```
|
98
103
|
|
@@ -106,7 +111,7 @@ Droplets can also take a hash with the following options:
|
|
106
111
|
`#stream` can also take an optional stream adapter (any object that implements `#put_records`) to only stream to that adapter. For example:
|
107
112
|
|
108
113
|
```ruby
|
109
|
-
droplet = Monsoon::Droplet.new(
|
114
|
+
droplet = Monsoon::Droplet.new({stream: 'transcodes', event: 'download_complete', media_id: 5, url: 'example.com/my_movie.mp4'}, {versioning: :skip})
|
110
115
|
droplet.stream(YourNewAdapter.new)
|
111
116
|
```
|
112
117
|
|
data/lib/monsoon.rb
CHANGED
@@ -5,6 +5,7 @@ require 'monsoon/droplet'
|
|
5
5
|
module Monsoon
|
6
6
|
@@streams = []
|
7
7
|
@@versions_schema = {}
|
8
|
+
@@default_stream = nil
|
8
9
|
|
9
10
|
def self.streams
|
10
11
|
@@streams
|
@@ -21,4 +22,12 @@ module Monsoon
|
|
21
22
|
def self.versions_schema=(versions_schema)
|
22
23
|
@@versions_schema = versions_schema
|
23
24
|
end
|
25
|
+
|
26
|
+
def self.default_stream
|
27
|
+
@@default_stream
|
28
|
+
end
|
29
|
+
|
30
|
+
def self.default_stream=(stream_name)
|
31
|
+
@@default_stream = stream_name
|
32
|
+
end
|
24
33
|
end
|
data/lib/monsoon/droplet.rb
CHANGED
@@ -6,21 +6,23 @@ module Monsoon
|
|
6
6
|
attr_reader :data
|
7
7
|
|
8
8
|
# new - creates a droplet
|
9
|
-
# stream_name - name of the stream to write to (required)
|
10
9
|
# data - hash of data to write to stream (required)
|
11
10
|
# options - optional hash
|
12
11
|
# :versioning => can be :skip, :enforce, or nil
|
13
12
|
# :skip will ignore the versions schema and write the data exactly as passed
|
14
13
|
# :enforce will require using the schema and will write nothing if unable to version
|
15
14
|
# nil will try to write versioned droplets and fallback to raw data (default)
|
15
|
+
# :partition_key => the partition key to use for kinesis
|
16
16
|
#
|
17
17
|
# @example
|
18
|
-
# Monsoon::Droplet.new('
|
19
|
-
def initialize(
|
20
|
-
@stream_name =
|
18
|
+
# Monsoon::Droplet.new({stream: 'transcodes', event: 'download_complete', filename: 'my_movie.mp4', media_id: 5, url: 'example.com/my_movie.mp4'}, {versioning: :enforce})
|
19
|
+
def initialize(raw_data, options = {})
|
20
|
+
@stream_name = raw_data.delete(:stream) || raw_data.delete('stream') || Monsoon.default_stream
|
21
|
+
raise ArgumentError, "stream not specified" unless @stream_name
|
22
|
+
@event = raw_data[:event] || raw_data['event']
|
21
23
|
@options = options
|
22
24
|
@raw_data = raw_data
|
23
|
-
@data = VersionsSchema.new(@stream_name).get_droplets(@raw_data) unless @options[:versioning] == :skip
|
25
|
+
@data = VersionsSchema.new(@stream_name, @event).get_droplets(@raw_data) unless @options[:versioning] == :skip
|
24
26
|
@data = [@raw_data] if blank? && @options[:versioning] != :enforce
|
25
27
|
end
|
26
28
|
|
@@ -28,11 +30,11 @@ module Monsoon
|
|
28
30
|
# streamer - (optional) stream adapter instance to limit which adapter is used (adapter must implement `#put_records`)
|
29
31
|
#
|
30
32
|
# @example
|
31
|
-
# droplet = Monsoon::Droplet.new('
|
33
|
+
# droplet = Monsoon::Droplet.new({stream: 'transcodes', event: 'download_complete', filename: 'my_movie.mp4', media_id: 5, url: 'example.com/my_movie.mp4'})
|
32
34
|
# droplet.stream(Monsoon::Streams::Console.new)
|
33
35
|
def stream(streamer = false)
|
34
36
|
return if blank?
|
35
|
-
streamer ? streamer.put_records(@stream_name, @data) : Monsoon.streams.each {|s| stream(s) }
|
37
|
+
streamer ? streamer.put_records(@stream_name, @data, @options) : Monsoon.streams.each {|s| stream(s) }
|
36
38
|
end
|
37
39
|
|
38
40
|
def blank?
|
@@ -7,11 +7,11 @@ module Monsoon
|
|
7
7
|
@client = Aws::Kinesis::Client.new
|
8
8
|
end
|
9
9
|
|
10
|
-
def put_records(stream, records)
|
10
|
+
def put_records(stream, records, options = {})
|
11
11
|
data = records.map do |r|
|
12
12
|
{
|
13
13
|
data: JSON.generate(r),
|
14
|
-
partition_key: 'monsoon'
|
14
|
+
partition_key: options[:partition_key] || r['event'] || r[:event] || r['droplet_version'] || 'monsoon'
|
15
15
|
}
|
16
16
|
end
|
17
17
|
@client.put_records(records: data, stream_name: stream)
|
data/lib/monsoon/version.rb
CHANGED
@@ -1,15 +1,17 @@
|
|
1
1
|
module Monsoon
|
2
2
|
class VersionsSchema
|
3
3
|
attr_reader :stream
|
4
|
+
attr_reader :event
|
4
5
|
|
5
|
-
def initialize(stream)
|
6
|
+
def initialize(stream, event)
|
6
7
|
@stream = stream
|
8
|
+
@event = event
|
7
9
|
end
|
8
10
|
|
9
11
|
def get_droplets(record)
|
10
12
|
return if record.nil?
|
11
13
|
|
12
|
-
|
14
|
+
event_schema.map do |version, keys|
|
13
15
|
symbolized_keys = keys.map(&:to_sym)
|
14
16
|
# Only keep the record keys that are defined in this version's schema
|
15
17
|
versioned_record = record.select{|k,v| symbolized_keys.include?(k.to_sym)}
|
@@ -18,7 +20,8 @@ module Monsoon
|
|
18
20
|
# add versioning metadata
|
19
21
|
versioned_record.merge({
|
20
22
|
'droplet_version' => version.to_s,
|
21
|
-
'droplet_deprecated' => deprecated?(version)
|
23
|
+
'droplet_deprecated' => deprecated?(version),
|
24
|
+
'event' => @event
|
22
25
|
}) if versioned_record.keys.length == keys.length # make sure we got all the keys for this version
|
23
26
|
end.compact
|
24
27
|
end
|
@@ -28,13 +31,19 @@ module Monsoon
|
|
28
31
|
end
|
29
32
|
|
30
33
|
def versions
|
31
|
-
|
34
|
+
event_schema.keys
|
32
35
|
end
|
33
36
|
|
34
37
|
private
|
35
38
|
|
36
|
-
def
|
37
|
-
|
39
|
+
def stream_schema
|
40
|
+
return {} unless @stream
|
41
|
+
@stream_schema ||= Monsoon.versions_schema[@stream.to_sym] || Monsoon.versions_schema[@stream.to_s] || {}
|
42
|
+
end
|
43
|
+
|
44
|
+
def event_schema
|
45
|
+
return {} unless @event
|
46
|
+
@event_schema ||= stream_schema[@event.to_sym] || stream_schema[@event.to_s] || {}
|
38
47
|
end
|
39
48
|
end
|
40
49
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: monsoon-droplet
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.1.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Neil Gupta
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2016-
|
11
|
+
date: 2016-03-03 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: aws-sdk
|