yakc 0.1.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: aba05e366fbce78b59bfabd65a5dc506ff79db18
4
+ data.tar.gz: 52d04d092dc31ba17edbdeb9179e223357852ac4
5
+ SHA512:
6
+ metadata.gz: 293a6fed65d496861799724c8eb3053bd6967d68de902ff23fd218a75656783956c3f6598d3f017e3a29b0d76d69955e3c6f7c9fdc888c2b6e11887ef638598f
7
+ data.tar.gz: f950ee60dc9132490bf70d41dc1f6e84c3e5bd9b7141318aaa010dea800bdc2c83cda5d98babcb73caf69dfd48a9886769f2868cddcc9ea23b1e59ddfdade889
data/.gitignore ADDED
@@ -0,0 +1,9 @@
1
+ /.bundle/
2
+ /.yardoc
3
+ /Gemfile.lock
4
+ /_yardoc/
5
+ /coverage/
6
+ /doc/
7
+ /pkg/
8
+ /spec/reports/
9
+ /tmp/
data/.travis.yml ADDED
@@ -0,0 +1,5 @@
1
+ sudo: false
2
+ language: ruby
3
+ rvm:
4
+ - 2.3.1
5
+ before_install: gem install bundler -v 1.12.5
data/Gemfile ADDED
@@ -0,0 +1,4 @@
1
+ source 'https://rubygems.org'
2
+
3
+ # Specify your gem's dependencies in yakc.gemspec
4
+ gemspec
data/LICENSE.txt ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2016 Greg
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,183 @@
1
+ # YAKC: Yet another Kafka (0.8) consumer
2
+
3
+ YAKC is a generic Kavka 0.8 consumer based on the now-dead Poseidon (i know, i know). It will listen to as many topics as you specify and hand them off via a handler to consumer classes.
4
+
5
+ ## Installation
6
+
7
+ Add this line to your application's Gemfile:
8
+
9
+ ```ruby
10
+ gem 'yakc'
11
+ ```
12
+
13
+ And then execute:
14
+
15
+ $ bundle
16
+
17
+ Or install it yourself as:
18
+
19
+ $ gem install yakc
20
+
21
+ ## Usage
22
+
23
+ There are 2 main componets:
24
+
25
+ ### Message Handler
26
+
27
+ This is the bit of code that handles what to do with the messages once they are received. There are 2 stages to this process:
28
+
29
+ 1. The message is parsed using *your* message parser(inherited from the `YAKC::Message` class) that does the parsing and validity checking.
30
+ 2. The parsed message payload is broadcast to the system. You can specify your own publisher, but by default the handler will use [Yeller](http://www.github.com/gaorlov/yeller). It will broadcast the message with the key: "topic::event"
31
+
32
+ To set it up:
33
+
34
+ ```ruby
35
+ handler = YAKC::MessageBroadcaster.new publisher: MyBroadcaster, message_class: MyMessageClass
36
+ # or, if you are okay with Yeller
37
+ handler = YAKC::MessageBroadcaster.new message_class: MyMessageClass
38
+ ```
39
+
40
+ And now you're ready to init the [reader](#reader)
41
+
42
+ #### Publisher Interface
43
+
44
+ If you don't like Yeller, or want something that can talk cross-process, you can implement your own.
45
+
46
+ The publisher interface is pretty simple: it has to implement
47
+ * `broadcast( message, topic )` : This is the function that handles where the messages go.
48
+
49
+ #### Message Interface
50
+
51
+ The message parser needs to implement:
52
+
53
+ 1. `parse( raw_message )` : This converts the raw Kafka data to the format of your choice
54
+ 2. `broadcastable?` : This determines whether the message is valid and shoud be broadcast.
55
+ 3: `event` : The name of the picked up event. This is the name that gets broadcast
56
+
57
+ For example if your messages are encoded in Avro and look loosely like:
58
+ ```json
59
+ { "event": {"name":"myEventName",
60
+ "timestamp":"00:00:00:12/12/12"}},
61
+ "my_field":"value",
62
+ // etc
63
+ }
64
+ ```
65
+
66
+ Your message parser class would look something like
67
+
68
+ ```ruby
69
+ class AvroMessage < YAKC::Message
70
+
71
+ def broadcastable?
72
+ # an event is probably okay to transmit if we can extract its name
73
+ event["name"]
74
+ end
75
+
76
+ def event
77
+ @payload["event"] || {}
78
+ end
79
+
80
+ protected
81
+
82
+ def parse( message )
83
+ data = StringIO.new(message.value)
84
+ msg = Avro::DataFile::Reader.new(data, Avro::IO::DatumReader.new)
85
+
86
+ rescue Avro::DataFile::DataFileError => e
87
+ Rails.logger.error e
88
+ {}
89
+ end
90
+ end
91
+
92
+ ```
93
+
94
+
95
+ ### Reader
96
+
97
+ The reader does(surprise) the reading and pushes the read rad messages to the handler, which you have to specify.
98
+
99
+ It implements:
100
+
101
+ * `read` : an infinite loop that consumes messages on all the specified topics (see [setup](#setup) below) and sends them to the handler
102
+
103
+ Here's how you would use it:
104
+
105
+ ```ruby
106
+ handler = YAKC::MessageBroadcaster.new message_class: AvroMessage
107
+ reader = YAKC::Reader.new message_handler: handler
108
+
109
+ reader.read
110
+ ```
111
+
112
+ ## Setup
113
+
114
+ And now for the full setup. You will need to specify the `zookeepers`; the Kafka `brokers`; the `app` and `suffix`, which are used to generate the consumer group name; the topic list; and a logger.
115
+
116
+ There are 2 ways of doing this. You can either set those up as ENV vars ("ZOOKEEPERS"(comma separated list), "BROKERS"(comma separated list), "APP", "SUFFIX", "TOPICS") and set up the logger by hand, or, you can do it in ruby, like:
117
+
118
+ ```ruby
119
+ YAKC.configure do |config|
120
+ config.logger = Rails.logger
121
+ config.zookeepers = ["localhost:9092"]
122
+ config.brokers = ["localhost:2181"]
123
+ config.app = "MyApp"
124
+ config.suffix = Rails.env
125
+ config.topics = ["clickstream", "logs", "exceptions"] # whatever you're listening for
126
+ end
127
+ ```
128
+
129
+ ## Example
130
+
131
+ Here's what a full experience would look like:
132
+
133
+ The reader would look like
134
+ ```ruby
135
+ # in your initializer
136
+ YAKC.configure do |config|
137
+ # we'll assume the rest are set up in the env
138
+ config.logger = Rails.logger
139
+ end
140
+ ```
141
+
142
+ In your reader job
143
+
144
+ ```ruby
145
+ handler = YAKC::MessageBroadcaster.new message_class: AvroMessage
146
+ reader = YAKC::Reader.new message_handler: handler
147
+
148
+ reader.read
149
+ ```
150
+
151
+ And the consumers would listen to the events
152
+
153
+ Let's say you have an app that listens to exceptions that we pass around in kafka. It then stores them in a DB and passesthem off to Honeybadger. Your `Exception` model could do something like
154
+
155
+ ```ruby
156
+ class Exception < ActiveRecord::Base
157
+ include Yeller::Subscribable
158
+
159
+ # we don't care about the event type, so we subscribe to "exception::.*"
160
+ subscribe with: :from_kafka_event, to: "exception::.*"
161
+
162
+ def self.from_kafka_event( message )
163
+ create message
164
+ Honeybadger.notify message
165
+ end
166
+ end
167
+ ```
168
+
169
+ ## Development
170
+
171
+ After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
172
+
173
+ To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
174
+
175
+ ## Contributing
176
+
177
+ Bug reports and pull requests are welcome on GitHub at https://github.com/gaorlov/yakc.
178
+
179
+
180
+ ## License
181
+
182
+ The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
183
+
data/Rakefile ADDED
@@ -0,0 +1,10 @@
1
+ require "bundler/gem_tasks"
2
+ require "rake/testtask"
3
+
4
+ Rake::TestTask.new(:test) do |t|
5
+ t.libs << "test"
6
+ t.libs << "lib"
7
+ t.test_files = FileList['test/**/*_test.rb']
8
+ end
9
+
10
+ task :default => :test
data/bin/console ADDED
@@ -0,0 +1,14 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require "bundler/setup"
4
+ require "yakc"
5
+
6
+ # You can add fixtures and/or initialization code here to make experimenting
7
+ # with your gem easier. You can also use a different console, if you like.
8
+
9
+ # (If you use this, don't forget to add pry to your Gemfile!)
10
+ # require "pry"
11
+ # Pry.start
12
+
13
+ require "irb"
14
+ IRB.start
data/bin/setup ADDED
@@ -0,0 +1,8 @@
1
+ #!/usr/bin/env bash
2
+ set -euo pipefail
3
+ IFS=$'\n\t'
4
+ set -vx
5
+
6
+ bundle install
7
+
8
+ # Do any other automated setup that you need to do here
@@ -0,0 +1,14 @@
1
+ module YAKC
2
+ class Configuration
3
+ attr_accessor :zookeepers, :brokers, :app, :suffix, :topics, :logger
4
+
5
+ def initialize
6
+ logger = Logger.new(STDOUT)
7
+ brokers = ENV.fetch("BROKERS", "localhost:9092").split(",")
8
+ zookeepers = ENV.fetch("ZOOKEEPERS", "localhost:2181").split(",")
9
+ app = ENV["APP"]
10
+ suffix = ENV["SUFFIX"]
11
+ topics = ENV["TOPICS"]
12
+ end
13
+ end
14
+ end
@@ -0,0 +1,10 @@
1
+ module YAKC
2
+ module Instrumenter
3
+ class FallthroughInstrumenter
4
+
5
+ def instrument( message )
6
+ yield
7
+ end
8
+ end
9
+ end
10
+ end
@@ -0,0 +1,23 @@
1
+ module YAKC
2
+ class Message
3
+ attr_reader :payload
4
+
5
+ def initialize( message )
6
+ @payload = parse( message )
7
+ end
8
+
9
+ def broadcastable?
10
+ # implement me
11
+ end
12
+
13
+ def event
14
+ # implement me
15
+ end
16
+
17
+ protected
18
+
19
+ def parse( message )
20
+ message
21
+ end
22
+ end
23
+ end
@@ -0,0 +1,29 @@
1
+ module YAKC
2
+ class MessageBroadcaster
3
+ attr_accessor :publisher, :message_class, :instrumenter
4
+
5
+ def initialize( publisher: Yeller, instrumenter: FallthroughInstrumenter, message_parser: )
6
+ @publisher = publisher
7
+ @message_class = message_parser
8
+ @instrumenter = instrumenter.new
9
+ raise "MessageBroadcaster must have a valid message class" unless @message_class
10
+ end
11
+
12
+ def handle( topic, message )
13
+ msg = @message_class.new( message )
14
+
15
+ @instrumenter.instrument( msg ) do
16
+ if msg.broadcastable?
17
+ # broadcast the specific topic event
18
+ @publisher.broadcast msg.payload, broadcast_key( topic, msg.event )
19
+ end
20
+ end
21
+ end
22
+
23
+ private
24
+
25
+ def broadcast_key( topic, event )
26
+ "#{topic}::#{event}"
27
+ end
28
+ end
29
+ end
@@ -0,0 +1,45 @@
1
+ module YAKC
2
+ class Reader
3
+ attr_reader :message_handler, :terminated
4
+
5
+ def initialize( message_handler: )
6
+ @message_handler = message_handler
7
+ @config = YAKC.configuration
8
+
9
+ raise KeyError, "YAKC::Reader initialized without a message handler. Please specify one so that your receives messages don't end up on the floor. For more info, go to: https://github.com/gaorlov/yakc#message-handler" unless message_handler
10
+
11
+ Signal.trap("INT") do
12
+ @terminated = true
13
+ end
14
+ end
15
+
16
+ def read
17
+ loop do
18
+ consumers.map do |consumer|
19
+ consumer.fetch do |partition, bulk|
20
+ bulk.each do |message|
21
+ message_handler.handle topic, message
22
+ end
23
+ end
24
+ return if terminated
25
+ end
26
+ end
27
+ rescue => e
28
+ YACK.logger.error e
29
+ retry
30
+ end
31
+
32
+ private
33
+
34
+ def consumers
35
+ @consumers ||= topics.map do |topic|
36
+ Poseidon::ConsumerGroup.new(
37
+ "#{app}-#{topic}-consumer-#{suffix}",
38
+ @config.brokers,
39
+ @config.zookeepers,
40
+ topic,
41
+ {})
42
+ end
43
+ end
44
+ end
45
+ end
@@ -0,0 +1,3 @@
1
+ module Yakc
2
+ VERSION = "0.1.0"
3
+ end
data/lib/yakc.rb ADDED
@@ -0,0 +1,26 @@
1
+ require "yakc/version"
2
+ require 'active_support'
3
+ require 'active_support/core_ext/module/delegation'
4
+ require 'yeller'
5
+
6
+ module YAKC
7
+ autoload :Configuration, 'yakc/configuration'
8
+ autoload :FallthroughInstrumenter, 'yakc/fallthorugh_instrumenter'
9
+ autoload :MessageBroadcaster, 'yakc/message_broadcaster'
10
+ autoload :Reader, 'yakc/reader'
11
+ autoload :Message, 'yakc/message'
12
+
13
+ class << self
14
+ attr_writer :configuration
15
+ end
16
+
17
+ def self.configuration
18
+ @configuration ||= Configuration.new
19
+ end
20
+
21
+ def self.configure
22
+ yield configuration
23
+ end
24
+
25
+ delegate :logger, to: :configuration
26
+ end
data/yakc.gemspec ADDED
@@ -0,0 +1,29 @@
1
+ # coding: utf-8
2
+ lib = File.expand_path('../lib', __FILE__)
3
+ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
+ require 'yakc/version'
5
+
6
+ Gem::Specification.new do |spec|
7
+ spec.name = "yakc"
8
+ spec.version = Yakc::VERSION
9
+ spec.authors = ["Greg"]
10
+ spec.email = ["greg@avvo.com"]
11
+
12
+ spec.summary = "Multitopic Poseidon-base Kafaka consumer"
13
+ spec.homepage = "http://www.github.com/gaorlov/yakc"
14
+ spec.license = "MIT"
15
+
16
+ spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
17
+ spec.bindir = "exe"
18
+ spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
19
+ spec.require_paths = ["lib"]
20
+
21
+ spec.add_dependency "yeller"
22
+ spec.add_dependency "poseidon_cluster", "0.3.2.avvo3"
23
+ spec.add_dependency "activesupport"
24
+
25
+ spec.add_development_dependency "bundler", "~> 1.12"
26
+ spec.add_development_dependency "rake", "~> 10.0"
27
+ spec.add_development_dependency "minitest", "~> 5.0"
28
+ spec.add_development_dependency "simplecov"
29
+ end
metadata ADDED
@@ -0,0 +1,159 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: yakc
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ platform: ruby
6
+ authors:
7
+ - Greg
8
+ autorequire:
9
+ bindir: exe
10
+ cert_chain: []
11
+ date: 2016-12-30 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: yeller
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - ">="
18
+ - !ruby/object:Gem::Version
19
+ version: '0'
20
+ type: :runtime
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - ">="
25
+ - !ruby/object:Gem::Version
26
+ version: '0'
27
+ - !ruby/object:Gem::Dependency
28
+ name: poseidon_cluster
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - '='
32
+ - !ruby/object:Gem::Version
33
+ version: 0.3.2.avvo3
34
+ type: :runtime
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - '='
39
+ - !ruby/object:Gem::Version
40
+ version: 0.3.2.avvo3
41
+ - !ruby/object:Gem::Dependency
42
+ name: activesupport
43
+ requirement: !ruby/object:Gem::Requirement
44
+ requirements:
45
+ - - ">="
46
+ - !ruby/object:Gem::Version
47
+ version: '0'
48
+ type: :runtime
49
+ prerelease: false
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - ">="
53
+ - !ruby/object:Gem::Version
54
+ version: '0'
55
+ - !ruby/object:Gem::Dependency
56
+ name: bundler
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - "~>"
60
+ - !ruby/object:Gem::Version
61
+ version: '1.12'
62
+ type: :development
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - "~>"
67
+ - !ruby/object:Gem::Version
68
+ version: '1.12'
69
+ - !ruby/object:Gem::Dependency
70
+ name: rake
71
+ requirement: !ruby/object:Gem::Requirement
72
+ requirements:
73
+ - - "~>"
74
+ - !ruby/object:Gem::Version
75
+ version: '10.0'
76
+ type: :development
77
+ prerelease: false
78
+ version_requirements: !ruby/object:Gem::Requirement
79
+ requirements:
80
+ - - "~>"
81
+ - !ruby/object:Gem::Version
82
+ version: '10.0'
83
+ - !ruby/object:Gem::Dependency
84
+ name: minitest
85
+ requirement: !ruby/object:Gem::Requirement
86
+ requirements:
87
+ - - "~>"
88
+ - !ruby/object:Gem::Version
89
+ version: '5.0'
90
+ type: :development
91
+ prerelease: false
92
+ version_requirements: !ruby/object:Gem::Requirement
93
+ requirements:
94
+ - - "~>"
95
+ - !ruby/object:Gem::Version
96
+ version: '5.0'
97
+ - !ruby/object:Gem::Dependency
98
+ name: simplecov
99
+ requirement: !ruby/object:Gem::Requirement
100
+ requirements:
101
+ - - ">="
102
+ - !ruby/object:Gem::Version
103
+ version: '0'
104
+ type: :development
105
+ prerelease: false
106
+ version_requirements: !ruby/object:Gem::Requirement
107
+ requirements:
108
+ - - ">="
109
+ - !ruby/object:Gem::Version
110
+ version: '0'
111
+ description:
112
+ email:
113
+ - greg@avvo.com
114
+ executables: []
115
+ extensions: []
116
+ extra_rdoc_files: []
117
+ files:
118
+ - ".gitignore"
119
+ - ".travis.yml"
120
+ - Gemfile
121
+ - LICENSE.txt
122
+ - README.md
123
+ - Rakefile
124
+ - bin/console
125
+ - bin/setup
126
+ - lib/yakc.rb
127
+ - lib/yakc/configuration.rb
128
+ - lib/yakc/fallthrough_instrumenter.rb
129
+ - lib/yakc/message.rb
130
+ - lib/yakc/message_broadcaster.rb
131
+ - lib/yakc/reader.rb
132
+ - lib/yakc/version.rb
133
+ - yakc.gemspec
134
+ homepage: http://www.github.com/gaorlov/yakc
135
+ licenses:
136
+ - MIT
137
+ metadata: {}
138
+ post_install_message:
139
+ rdoc_options: []
140
+ require_paths:
141
+ - lib
142
+ required_ruby_version: !ruby/object:Gem::Requirement
143
+ requirements:
144
+ - - ">="
145
+ - !ruby/object:Gem::Version
146
+ version: '0'
147
+ required_rubygems_version: !ruby/object:Gem::Requirement
148
+ requirements:
149
+ - - ">="
150
+ - !ruby/object:Gem::Version
151
+ version: '0'
152
+ requirements: []
153
+ rubyforge_project:
154
+ rubygems_version: 2.5.1
155
+ signing_key:
156
+ specification_version: 4
157
+ summary: Multitopic Poseidon-base Kafaka consumer
158
+ test_files: []
159
+ has_rdoc: