railway-ipc 2.1.0 → 4.0.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 3b65c6b026a85e5cd4ae6693711ec05d9efe714517c2a05c3bfd7111bde92eab
4
- data.tar.gz: 944093374956d0ad82460ebbddb61f952d3ac70dbdb8294f032dd63ee2e0df9e
3
+ metadata.gz: 7f349dc76232b1cef6492bb2dc09cdbd58bf2ad22be64bfd1771e4e6d7192b16
4
+ data.tar.gz: 3e04b076735ca1bcc8d1cc376925311b1547b8b13a554691726f348a62625cfc
5
5
  SHA512:
6
- metadata.gz: 7fad19c0a2c6bc99a7e56242eaa70b6c5f449d9c641372cc4088cc5aaba1f8ac3cfcc45f6e8b003c34713c65342b676114dd7ada579ed8d8bf3b31a94838d614
7
- data.tar.gz: 27e95ff1100792b27e07414ee681062ceb54f12c7c31308e022045184031c15793f5f187b9dfbe51746ffe0bd4cb4ab0bf1d81163cb04126598fba15d720a6ec
6
+ metadata.gz: d0dd3263f9ca6c57da0fd98c25b720f1f7bad7ec87bb4c34cd110d51824b350ca255d9a467170d420aeb323faff081f03ea62c18c080f3a52fd72f0c687be49a
7
+ data.tar.gz: 74d6f196446603d93d42050e2c672ccac22ec49cedbc2d026cf8f8f61eb0bc4d6537b179ad2d60cd37426ad6b60ab9e428d40b21f69ce21426cc86cbeefaacaf
data/.gitignore CHANGED
@@ -4,10 +4,14 @@
4
4
  /coverage/
5
5
  /doc/
6
6
  /pkg/
7
+ /rake
7
8
  /spec/reports/
8
9
  /tmp/
9
10
  Gemfile.lock
10
11
 
12
+ # built gems
13
+ railway-ipc-*.gem
14
+
11
15
  # rspec failure tracking
12
16
  .rspec_status
13
17
 
@@ -0,0 +1 @@
1
+ 2.6.5
@@ -10,6 +10,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
10
10
  ### Removed
11
11
  ### Fixed
12
12
 
13
+ ## [4.0.0] - 2021-01-11
14
+ ### Added
15
+ * JSON decoder for consumers that can handle JSON encoded Protobufs. Note that the publishers do not (yet) have the option of encoding the messages as JSON.
16
+
17
+ ### Changed
18
+ * (Breaking Change) Rename `Consumer#work` to `Consumer#work_with_params`. This was necessary so that we can support specifying different message encodings via metadata in the future. If the message encoding cannot be determined from the message metadata fall back to a default decoder (binary protobufs).
19
+
20
+ ### Fixed
21
+ * `./bin/console` script was broken because Pry wasn't a dependency; added Pry as a development dependency only.
22
+
23
+ ## [3.0.0] - 2020-12-07
24
+ ### Changed
25
+ * Consumers _will no longer crash_ when an exception is raised. Instead, consumers will move the message that caused the exception to a single dead-letter exchange called 'ipc:errors'. Railway will configure the dead-letter exchange automatically.
26
+
27
+ ## [2.2.2] - 2020-11-20
28
+ ### Fixed
29
+ * Fixed Publisher class channel leak. Channels were being created on each instantiation of a Publisher instead of being re-used.
30
+
31
+ ## [2.2.1] - 2020-10-20
32
+ ### Added
33
+ * Logging to indicate when options passed via `listen_to`/`from_queue` are overriding the defaults.
34
+
35
+ ## [2.2.0] - 2020-10-20
36
+ ### Added
37
+ * The ability to configure workers to handle different workloads via `rake railway_ipc::consumers:spawn`
38
+
39
+ ## [2.1.0] - 2020-10-19
40
+ ### Added
41
+ * :options parameter to `listen_to` which passes keys along to Sneaker's `from_queue` method.
42
+
13
43
  ## [2.0.3] - 2020-09-02
14
44
  ### Fixed
15
45
  * Fix RPC server. RPC servers need to conform to the Sneaker worker API (i.e. their initializers need to be able to accept queue name / pool and they require a `stop` method.
@@ -66,7 +96,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
66
96
  ### Added
67
97
  - Correlation ID and message UUID are auto generated for messages for IDs are not passed in [#23](https://github.com/learn-co/railway_ipc_gem/pull/23)
68
98
 
69
- [Unreleased]: https://github.com/learn-co/railway_ipc_gem/compare/v2.0.3...HEAD
99
+ [Unreleased]: https://github.com/learn-co/railway_ipc_gem/compare/v4.0.0...HEAD
100
+ [4.0.0]: https://github.com/learn-co/railway_ipc_gem/compare/v3.0.0...v4.0.0
101
+ [3.0.0]: https://github.com/learn-co/railway_ipc_gem/compare/v2.2.2...v3.0.0
102
+ [2.2.2]: https://github.com/learn-co/railway_ipc_gem/compare/v2.2.1...v2.2.2
103
+ [2.2.1]: https://github.com/learn-co/railway_ipc_gem/compare/v2.2.0...v2.2.1
104
+ [2.2.0]: https://github.com/learn-co/railway_ipc_gem/compare/v2.1.0...v2.2.0
105
+ [2.1.0]: https://github.com/learn-co/railway_ipc_gem/compare/v2.0.3...v2.1.0
70
106
  [2.0.3]: https://github.com/learn-co/railway_ipc_gem/compare/v2.0.2...v2.0.3
71
107
  [2.0.2]: https://github.com/learn-co/railway_ipc_gem/compare/v2.0.1...v2.0.2
72
108
  [2.0.1]: https://github.com/learn-co/railway_ipc_gem/compare/v2.0.0...v2.0.1
data/README.md CHANGED
@@ -52,6 +52,26 @@ Then, run your consumers
52
52
  bundle exec rake railway_ipc:consumers:start CONSUMERS=YourConsumer,YourOtherConsumer
53
53
  ```
54
54
 
55
+ You may also configure your consumers more granularly using:
56
+
57
+ `./config/sneaker_worker_groups.yml`
58
+ ```yaml
59
+ HighPriority:
60
+ classes: YourConsumer,YourOtherConsumer
61
+ workers: 5
62
+ LowPriority:
63
+ classes: YourThirdConsumer
64
+ workers: 1
65
+ ```
66
+
67
+ ```bash
68
+ bundle exec rake railway_ipc:consumers:spawn
69
+ ```
70
+
71
+ By default, `spawn` will map to `./config/sneaker_worker_groups.yml` but you can override it by using `WORKER_GROUP_CONFIG`.
72
+
73
+ See the [Sneaker Documentation](https://github.com/jondot/sneakers/wiki/Handling-different-workloads) for more information.
74
+
55
75
  # Request/Response
56
76
 
57
77
  Define your server, client and responder. Docs coming soon.
@@ -2,9 +2,10 @@
2
2
 
3
3
  require 'railway_ipc/version'
4
4
  require 'sneakers'
5
+ require 'sneakers/spawner'
5
6
  require 'bunny'
6
7
  require 'active_record'
7
- require 'railway_ipc/version'
8
+ require 'singleton'
8
9
  require 'railway_ipc/logger'
9
10
  require 'railway_ipc/unhandled_message_error'
10
11
  require 'railway_ipc/response'
@@ -13,7 +14,9 @@ require 'railway_ipc/unknown_message.pb'
13
14
  require 'railway_ipc/rabbitmq/adapter'
14
15
  require 'railway_ipc/handler'
15
16
  require 'railway_ipc/handler_store'
17
+ require 'railway_ipc/message_decoders'
16
18
  require 'railway_ipc/incoming_message'
19
+ require 'railway_ipc/connection_manager'
17
20
  require 'railway_ipc/publisher'
18
21
  require 'railway_ipc/responder'
19
22
  require 'railway_ipc/rpc/rpc'
@@ -29,12 +32,16 @@ module RailwayIpc
29
32
  Rake::Task['sneakers:run'].invoke
30
33
  end
31
34
 
32
- def self.configure(log_device=STDOUT, level=::Logger::INFO, log_formatter=nil)
35
+ def self.spawn
36
+ Sneakers::Spawner.spawn
37
+ end
38
+
39
+ def self.configure(log_device=$stdout, level=::Logger::INFO, log_formatter=nil)
33
40
  @logger = RailwayIpc::Logger.new(log_device, level, log_formatter)
34
41
  end
35
42
 
36
43
  def self.logger
37
- @logger || RailwayIpc::Logger.new(STDOUT)
44
+ @logger || RailwayIpc::Logger.new($stdout)
38
45
  end
39
46
 
40
47
  def self.bunny_connection
@@ -0,0 +1,52 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'singleton'
4
+
5
+ module RailwayIpc
6
+ # RabbitMQ connection manager. Ensures there is a single RabbitMQ
7
+ # connection and channel per thread, which prevents channel leaks.
8
+ #
9
+ class ConnectionManager
10
+ include Singleton
11
+
12
+ def initialize
13
+ establish_connection
14
+ end
15
+
16
+ def establish_connection
17
+ @connection = Bunny.new(
18
+ host: settings[:host],
19
+ user: settings[:user],
20
+ pass: settings[:pass],
21
+ port: settings[:port],
22
+ vhost: settings[:vhost] || '/',
23
+ logger: RailwayIpc.logger
24
+ )
25
+ @connection.start
26
+ @channel = @connection.create_channel
27
+
28
+ @connection
29
+ end
30
+
31
+ def channel
32
+ return @channel if connected?
33
+
34
+ establish_connection
35
+ @channel
36
+ end
37
+
38
+ def connected?
39
+ @connection&.connected? && @channel&.open?
40
+ end
41
+
42
+ private
43
+
44
+ def amqp_url
45
+ @amqp_url ||= ENV.fetch('RAILWAY_RABBITMQ_CONNECTION_URL', 'amqp://guest:guest@localhost:5672')
46
+ end
47
+
48
+ def settings
49
+ @settings ||= AMQ::Settings.parse_amqp_url(amqp_url)
50
+ end
51
+ end
52
+ end
@@ -5,6 +5,8 @@ module RailwayIpc
5
5
  include Sneakers::Worker
6
6
 
7
7
  def self.inherited(base)
8
+ super
9
+
8
10
  base.instance_eval do
9
11
  def handlers
10
12
  @handlers ||= RailwayIpc::HandlerStore.new
@@ -12,14 +14,27 @@ module RailwayIpc
12
14
  end
13
15
  end
14
16
 
17
+ # rubocop:disable Metrics/MethodLength
15
18
  def self.listen_to(queue:, exchange:, options: {})
19
+ unless options.empty?
20
+ RailwayIpc.logger.info(
21
+ "Overriding configuration for #{queue} with new options",
22
+ feature: 'railway_ipc_consumer',
23
+ options: options
24
+ )
25
+ end
26
+
16
27
  from_queue queue, {
17
28
  exchange: exchange,
18
29
  durable: true,
19
30
  exchange_type: :fanout,
31
+ arguments: {
32
+ 'x-dead-letter-exchange' => 'ipc:errors'
33
+ },
20
34
  connection: RailwayIpc.bunny_connection
21
35
  }.merge(options)
22
36
  end
37
+ # rubocop:enable Metrics/MethodLength
23
38
 
24
39
  def self.handle(message_type, with:)
25
40
  handlers.register(message: message_type, handler: with)
@@ -41,8 +56,14 @@ module RailwayIpc
41
56
  queue.opts[:exchange]
42
57
  end
43
58
 
44
- def work(payload)
45
- message = RailwayIpc::IncomingMessage.new(payload)
59
+ # REFACTOR: Long term we should think about not leaking Sneakers
60
+ # methods as part of Railway's public API since clients can (and do)
61
+ # override them. -BN
62
+ def work_with_params(payload, _delivery_info, metadata)
63
+ message_format = metadata.fetch('headers', {})
64
+ .fetch('message_format', 'protobuf_binary')
65
+
66
+ message = RailwayIpc::IncomingMessage.new(payload, message_format: message_format)
46
67
  RailwayIpc::ProcessIncomingMessage.call(self, message)
47
68
  ack!
48
69
  rescue StandardError => e
@@ -54,7 +75,7 @@ module RailwayIpc
54
75
  error: e.class,
55
76
  payload: payload
56
77
  )
57
- raise e
78
+ reject!
58
79
  end
59
80
 
60
81
  def get_handler(type)
@@ -2,9 +2,10 @@
2
2
 
3
3
  module RailwayIpc
4
4
  class IncomingMessage
5
- attr_reader :type, :payload, :parsed_payload, :errors
5
+ attr_reader :type, :message_format, :payload, :parsed_payload, :errors
6
6
 
7
- def initialize(payload)
7
+ def initialize(payload, message_format: nil)
8
+ @message_format = message_format
8
9
  @parsed_payload = JSON.parse(payload)
9
10
  @type = parsed_payload['type']
10
11
  @payload = payload
@@ -32,20 +33,23 @@ module RailwayIpc
32
33
  end
33
34
 
34
35
  def decoded
35
- @decoded ||=
36
- begin
37
- protobuf_msg = Base64.decode64(parsed_payload['encoded_message'])
38
- decoder = Kernel.const_get(type)
39
- decoder.decode(protobuf_msg)
40
- rescue Google::Protobuf::ParseError => e
41
- raise RailwayIpc::IncomingMessage::ParserError.new(e)
42
- rescue NameError
43
- RailwayIpc::Messages::Unknown.decode(protobuf_msg)
44
- end
36
+ @decoded ||= \
37
+ get_decoder(message_format).call(type, parsed_payload['encoded_message'])
45
38
  end
46
39
 
47
40
  def stringify_errors
48
41
  errors.values.join(', ')
49
42
  end
43
+
44
+ private
45
+
46
+ DEFAULT_DECODER = RailwayIpc::MessageDecoders::ProtobufBinaryDecoder
47
+
48
+ def get_decoder(name)
49
+ {
50
+ 'protobuf_binary' => RailwayIpc::MessageDecoders::ProtobufBinaryDecoder,
51
+ 'protobuf_json' => RailwayIpc::MessageDecoders::ProtobufJsonDecoder
52
+ }.fetch(name, DEFAULT_DECODER)
53
+ end
50
54
  end
51
55
  end
@@ -21,7 +21,7 @@ module RailwayIpc
21
21
  # logger = RailwayIpc::Logger.new(STDOUT, Logger::INFO, OjFormatter)
22
22
  #
23
23
  class Logger
24
- def initialize(device=STDOUT, level=::Logger::INFO, formatter=nil)
24
+ def initialize(device=$stdout, level=::Logger::INFO, formatter=nil)
25
25
  @logger = ::Logger.new(device)
26
26
  @logger.level = level
27
27
  @logger.formatter = formatter if formatter
@@ -32,7 +32,7 @@ module RailwayIpc
32
32
  data.merge!(feature: 'railway_ipc') unless data.key?(:feature)
33
33
  return logger.send(level, data.merge(message: message)) unless block
34
34
 
35
- data = message.merge(data) if message&.is_a?(Hash)
35
+ data = message.merge(data) if message.is_a?(Hash)
36
36
  data.merge!(message: block.call)
37
37
 
38
38
  # This is for compatability w/ Ruby's Logger. Ruby's Logger class
@@ -41,7 +41,7 @@ module RailwayIpc
41
41
  # is assumed to be the `progname`.
42
42
  #
43
43
  # https://github.com/ruby/logger/blob/master/lib/logger.rb#L471
44
- data.merge!(progname: message) if message&.is_a?(String)
44
+ data.merge!(progname: message) if message.is_a?(String)
45
45
  logger.send(level, data)
46
46
  end
47
47
  end
@@ -0,0 +1,34 @@
1
+ # frozen_string_literal: true
2
+
3
+ module RailwayIpc
4
+ module MessageDecoders
5
+ ProtobufBinaryDecoder = lambda do |type, encoded_message|
6
+ protobuf_msg = Base64.decode64(encoded_message)
7
+ protobuf_klass = Kernel.const_get(type)
8
+ protobuf_klass.decode(protobuf_msg)
9
+ rescue Google::Protobuf::ParseError => e
10
+ raise RailwayIpc::IncomingMessage::ParserError.new(e)
11
+ rescue NameError
12
+ RailwayIpc::Messages::Unknown.decode(protobuf_msg)
13
+ end
14
+
15
+ ProtobufJsonDecoder = lambda do |type, message_hash|
16
+ protobuf_klass = Kernel.const_get(type)
17
+ protobuf_klass.new(message_hash)
18
+ rescue ArgumentError => e
19
+ raise RailwayIpc::IncomingMessage::ParserError.new(e)
20
+ rescue NameError
21
+ # NOTE: I didn't realize this until I made this ProtobufJsonDecoder, but
22
+ # the ProtobufBinaryDecoder will ignore any unknown keys -- which is
23
+ # probably not what we want. I'm coding this the same way as the binary
24
+ # protobuf version for consistency, but we should re-think how we want to
25
+ # handle this situation. -BN
26
+ RailwayIpc::Messages::Unknown.new(
27
+ user_uuid: message_hash.fetch(:user_uuid, ''),
28
+ correlation_id: message_hash.fetch(:correlation_id, ''),
29
+ uuid: message_hash.fetch(:uuid, ''),
30
+ context: message_hash.fetch(:context, {})
31
+ )
32
+ end
33
+ end
34
+ end
@@ -1,7 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
- require 'singleton'
4
-
5
3
  module RailwayIpc
6
4
  class SingletonPublisher < Sneakers::Publisher
7
5
  include ::Singleton
@@ -53,19 +51,12 @@ module RailwayIpc
53
51
  end
54
52
 
55
53
  module RailwayIpc
56
- class Publisher < Sneakers::Publisher
54
+ class Publisher
57
55
  attr_reader :exchange_name, :message_store
58
56
 
59
57
  def initialize(opts={})
60
58
  @exchange_name = opts.fetch(:exchange_name)
61
59
  @message_store = opts.fetch(:message_store, RailwayIpc::PublishedMessage)
62
- connection = opts.fetch(:connection, nil)
63
- options = {
64
- exchange: exchange_name,
65
- connection: connection,
66
- exchange_type: :fanout
67
- }.compact
68
- super(options)
69
60
  end
70
61
 
71
62
  # rubocop:disable Metrics/AbcSize
@@ -75,7 +66,7 @@ module RailwayIpc
75
66
  RailwayIpc.logger.info('Publishing message', log_message_options(message))
76
67
 
77
68
  stored_message = message_store.store_message(exchange_name, message)
78
- super(RailwayIpc::Rabbitmq::Payload.encode(message))
69
+ exchange.publish(RailwayIpc::Rabbitmq::Payload.encode(message))
79
70
  rescue RailwayIpc::InvalidProtobuf => e
80
71
  RailwayIpc.logger.error('Invalid protobuf', log_message_options(message))
81
72
  raise e
@@ -88,8 +79,16 @@ module RailwayIpc
88
79
  end
89
80
  # rubocop:enable Metrics/AbcSize
90
81
 
82
+ def exchange
83
+ @exchange ||= channel.exchange(exchange_name, type: :fanout, durable: true, auto_delete: false, arguments: {})
84
+ end
85
+
91
86
  private
92
87
 
88
+ def channel
89
+ RailwayIpc::ConnectionManager.instance.channel
90
+ end
91
+
93
92
  def log_message_options(message)
94
93
  {
95
94
  feature: 'railway_ipc_publisher',
@@ -16,7 +16,7 @@ module RailwayIpc
16
16
  :port,
17
17
  :user
18
18
 
19
- def initialize(amqp_url: ENV['RAILWAY_RABBITMQ_CONNECTION_URL'], exchange_name:, queue_name: '', options: {})
19
+ def initialize(exchange_name:, amqp_url: ENV['RAILWAY_RABBITMQ_CONNECTION_URL'], queue_name: '', options: {})
20
20
  @queue_name = queue_name
21
21
  @exchange_name = exchange_name
22
22
  settings = AMQ::Settings.parse_amqp_url(amqp_url)
@@ -2,9 +2,15 @@
2
2
 
3
3
  namespace :railway_ipc do
4
4
  namespace :consumers do
5
+ desc 'Start consumers via explicitly defining them'
5
6
  task :start do
6
7
  ENV['WORKERS'] = ENV['CONSUMERS']
7
8
  RailwayIpc.start
8
9
  end
10
+
11
+ desc 'Start consumers via ./config/sneaker_worker_groups.yml'
12
+ task :spawn do
13
+ RailwayIpc.spawn
14
+ end
9
15
  end
10
16
  end
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module RailwayIpc
4
- VERSION = '2.1.0'
4
+ VERSION = '4.0.0'
5
5
  end
@@ -48,6 +48,7 @@ Gem::Specification.new do |spec|
48
48
  spec.add_development_dependency 'database_cleaner', '~> 1.7'
49
49
  spec.add_development_dependency 'listen', '~> 3.0.5'
50
50
  spec.add_development_dependency 'pg', '~> 0.18'
51
+ spec.add_development_dependency 'pry', '~> 0.13'
51
52
  spec.add_development_dependency 'rails', '~> 5.0.7'
52
53
  spec.add_development_dependency 'rspec-rails'
53
54
  spec.add_development_dependency 'shoulda-matchers', '~> 4.2'
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: railway-ipc
3
3
  version: !ruby/object:Gem::Version
4
- version: 2.1.0
4
+ version: 4.0.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - ''
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2020-10-19 00:00:00.000000000 Z
11
+ date: 2021-01-11 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
@@ -164,6 +164,20 @@ dependencies:
164
164
  - - "~>"
165
165
  - !ruby/object:Gem::Version
166
166
  version: '0.18'
167
+ - !ruby/object:Gem::Dependency
168
+ name: pry
169
+ requirement: !ruby/object:Gem::Requirement
170
+ requirements:
171
+ - - "~>"
172
+ - !ruby/object:Gem::Version
173
+ version: '0.13'
174
+ type: :development
175
+ prerelease: false
176
+ version_requirements: !ruby/object:Gem::Requirement
177
+ requirements:
178
+ - - "~>"
179
+ - !ruby/object:Gem::Version
180
+ version: '0.13'
167
181
  - !ruby/object:Gem::Dependency
168
182
  name: rails
169
183
  requirement: !ruby/object:Gem::Requirement
@@ -213,6 +227,7 @@ extensions: []
213
227
  extra_rdoc_files: []
214
228
  files:
215
229
  - ".gitignore"
230
+ - ".ruby-version"
216
231
  - CHANGELOG.md
217
232
  - CODE_OF_CONDUCT.md
218
233
  - Gemfile
@@ -225,6 +240,7 @@ files:
225
240
  - bin/setup
226
241
  - lib/railway_ipc.rb
227
242
  - lib/railway_ipc/Rakefile
243
+ - lib/railway_ipc/connection_manager.rb
228
244
  - lib/railway_ipc/consumer/consumer.rb
229
245
  - lib/railway_ipc/consumer/process_incoming_message.rb
230
246
  - lib/railway_ipc/errors.rb
@@ -232,6 +248,7 @@ files:
232
248
  - lib/railway_ipc/handler_store.rb
233
249
  - lib/railway_ipc/incoming_message.rb
234
250
  - lib/railway_ipc/logger.rb
251
+ - lib/railway_ipc/message_decoders.rb
235
252
  - lib/railway_ipc/models/consumed_message.rb
236
253
  - lib/railway_ipc/models/published_message.rb
237
254
  - lib/railway_ipc/publisher.rb