hermes_messenger_of_the_gods 2.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: ea4406994cf204a6e1f4d75e4f5214bce1551091beba8980d36bdf717da6c158
+  data.tar.gz: 724c1dd921922d02056144fe4c5c452901be2105bdfc0c62ecbbe8c8bda40e8c
+SHA512:
+  metadata.gz: 530736b7cd4f48e51a28771221ffe0d1e87bebd6ccf74c8c8e8671e0f41bd68a5e480150d0d9c5a768bbd01c78da78251b296e819f20e0bf1089b8b5b06e9cc8
+  data.tar.gz: e48f7e6eeee775133b6467a99a88d5971f4c6547f6dc3208866e1aa85cea639bbab1ced3bb9ff2203d7c6cff841bb99d607c4175ac41aec6b78bd1d3bc3ec26d

data/.drone.yml

@@ -0,0 +1,9 @@
+
+pipeline:
+  test:
+    image: getterminus/ruby-ci-image:2.6-je-20190205
+    group: test
+    secrets: [ CODECOV_TOKEN ]
+    commands:
+      - bundle --local
+      - bundle exec rspec

data/.gitignore

@@ -0,0 +1,12 @@
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+
+# rspec failure tracking
+.rspec_status
+.bundle

data/.rspec

@@ -0,0 +1,3 @@
+--order rand
+--format documentation
+--color

data/.ruby-version

@@ -0,0 +1 @@
+2.6.1

data/.travis.yml

@@ -0,0 +1,5 @@
+sudo: false
+language: ruby
+rvm:
+  - 2.1.7
+before_install: gem install bundler -v 1.14.6

data/Gemfile

@@ -0,0 +1,13 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in hermes-messenger-of-the-gods.gemspec
+gemspec
+
+gem 'protobuf3_fixer', git: 'https://github.com/GetTerminus/Protobuf3Fixer.git'
+gem 'grpc'
+
+group :test do
+  gem "simplecov"
+  gem "webmock"
+  gem "codeclimate-test-reporter", "~> 1.0.0"
+end

data/README.md

@@ -0,0 +1,464 @@
+# HermesMessengerOfTheGods
+[![CircleCI](https://circleci.com/gh/GetTerminus/hermes_messenger_of_the_gods.svg?style=svg&circle-token=9f757798d1f2d06b93ee2f560c0689cc71f1f96a)](https://circleci.com/gh/GetTerminus/hermes_messenger_of_the_gods)
+[![Code Climate](https://codeclimate.com/repos/58d16440cf31c32d1000041c/badges/bf27bdcafb886574909d/gpa.svg)](https://codeclimate.com/repos/58d16440cf31c32d1000041c/feed)
+[![Test Coverage](https://codeclimate.com/repos/58d16440cf31c32d1000041c/badges/bf27bdcafb886574909d/coverage.svg)](https://codeclimate.com/repos/58d16440cf31c32d1000041c/coverage)
+[![Issue Count](https://codeclimate.com/repos/58d16440cf31c32d1000041c/badges/bf27bdcafb886574909d/issue_count.svg)](https://codeclimate.com/repos/58d16440cf31c32d1000041c/feed)
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'hermes_messenger_of_the_gods', git: 'url_of_repo'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install hermes_messenger_of_the_gods
+
+## Usage
+
+### The Message
+
+A HMOTG Message includes ActiveModel::Base, so ou get alot of the joys related to that. Initialization can be setup with:
+
+#### Creating a Message
+When creating a message the most common parameters passed are the attributes
+which represent the content of the message.
+
+```ruby
+class Foo
+  include HermesMessengerOfTheGods::Concerns::Message
+  attr_accessor :name
+end
+
+Foo.new(name: 'Dude').name
+# => Dude
+```
+
+#### Validations
+HMOTG Supports the validation syntax provided by ActiveModel::Validations.
+
+```ruby
+class Foo
+  include HermesMessengerOfTheGods::Concerns::Message
+  attr_accessor :name
+  validates :name, presence: true
+end
+
+instance = Foo.new(name: '')
+instance.valid?
+# => false
+instance.errors
+# => {name: ["can't be blank"]}
+```
+
+### Configuring a Endpoint
+Endpoints are the recepients of message, and are typically configured per
+message. The [Endpoints](https://github.com/GetTerminus/hermes_messenger_of_the_gods#endpoints)
+section provides configuration details for each endpoint type.
+
+#### Setting Endpoints
+You may set a single endpoint for a mesage by assigning it to a class attribute
+endpoitns
+```ruby
+class Foo
+  include HermesMessengerOfTheGods::Concerns::Message
+
+  attr_accessor :name
+  validates :name, presence: true
+  self.endpoints = {default: sns_endpoint("sns:arn", jitter: false) }
+end
+```
+
+Alternatively you can set multiple endpoints using a hash syntax:
+
+```ruby
+class Foo
+  include HermesMessengerOfTheGods::Concerns::Message
+
+  attr_accessor :name
+  validates :name, presence: true
+  self.endpoints = {
+    receiver1: sns_endpoint("sns:arn", jitter: false),
+    receiver2: sns_endpoint("sns:arn", jitter: true),
+  }
+end
+```
+
+**Important:** When using multiple endpoints, if one endpoint of many fails,
+the entire message transmission is considered to have failed. Since we can't
+take a message back from an endpoint we can't do too much about it.
+
+#### Sending the message
+Messages can be dispatched to the respective endpoints using the `dispatch` or
+the error raising `dispatch!` method. By default the Message also includes a
+`to_message` method which returns a hash of the object's attributes.
+
+```ruby
+class Foo
+  include HermesMessengerOfTheGods::Concerns::Message
+
+  attr_accessor :name
+  validates :name, presence: true
+  self.endpoints = {
+    default: sns_endpoint("sns:arn", jitter: false),
+    other: sns_endpoint('...')
+  }
+end
+
+Foo.new(name: 'omg').dispatch!
+# => true
+```
+
+Dispatches to each endpoint is done in a thread to allow for better performance.
+
+##### Targeting an endpoint
+You may target only a specific endpoint by passing the `endpoints: ['']` option
+during message creation. Using the message above:
+`Foo.new(name: 'omg', endpoints: [:other]).dispatch!` would dispatch only to
+the other endpoint.
+
+
+#### Async message dispatch
+All messages contain a `dispatch_async(!)` option in addition to typical deferral.
+Using an async dispatch runs the dispatch in a thread. Since dispatches tend to
+be I/O heavy this is an ideal option if you do not care about capturing the
+return information or error status. You can check to determine how many messages
+are in-flight using the helper method
+`HermesMessengerOfTheGods.async_dispatches_in_progress` in the testing helpers
+a method named `wait_for_async_dispatches` is avaialbe which will wait for up
+to 5 seconds for all messages to be "delivered".
+
+### Endpoint responses
+When an endpoint reports successful transmission, the response is recorded in
+the `message.successes` hash. This hash is keyed to correspond to the
+configuration `endpoints` hash.
+
+### Endpoint Failures
+When an endpoint fails (raises an error) the final error raised will be stored
+in the `message.dispatch_errors` hash. This hash is keyed to correspond to the
+`endponts` configuration hash
+
+Failure Exceptions:
+* When all endpoints fail a `HermesMessengerOfTheGods::MessageDispatchTotalFailure` is raised
+* When only some endpoints fail a `HermesMessengerOfTheGods::MessageDispatchPartialFailure` is raised
+
+Both of these exepctions inherit from `HermesMessengerOfTheGods::MessageDispatchFailed`
+
+##  The Worker
+The worker provides a generic class to provide access for reading from
+endpoints. Only some endpoints actually provide the ability to read messages.
+
+### Configuration
+Workers can be configured with a single endpoint and a single MessageClass for
+deserialization.
+
+```ruby
+class MyWorker
+  include HermesMessengerOfTheGods::Concerns::Worker
+
+  self.endpoint = sns_endpoint("sns::arn", jitter: false)
+  self.deserialize_with = MessageType
+end
+```
+
+### Getting the work done
+When working off a messages coming from an endpoint, there are a few possible
+paths the message can take.
+
+1. You may define a `perform` method on the worker which takes the message
+   instance as the only argument.
+2. If a `perform` method is not defined on the worker, a `perform` method is
+   expected to be defined on the message itself. This method is called with no
+   arguments.
+3. If no `perform` method is found on the Worker or the Message an error is
+   raised.
+
+**Note:** A `perform` method on the worker is perfered to a perform method on
+the message.
+
+### Error Handling
+When a messsage fails to run, the `handle_failure` on the worker is invoked.
+This method is called with two arguments:
+
+1. The message itself
+2. The exception itself.
+
+By default this method simply passes the job and exception to the endpoint so
+it can be handled as appropiate. If you overload this method you probably want
+to ensure you call `super`.
+
+### Message Deserialization
+Messages coming off the endpoints need to be deserialized into the domain
+message. By default we expect you are deserializing into a
+HermesMessengerOfTheGods::Concerns::Message, but it can be anything really.
+
+In order to turn the received message into the Object we expect a class method
+`.from_message` to be defined. The HermesMessengerOfTheGods::Message superclass
+defines a basic one that expects all attributes present in the message to be
+`attr_accessor`s on the message object.
+
+The method used to create the message object can be overloaded by setting
+`deserialize_method` class variable of the worker.
+
+```ruby
+class MessageType
+  include HermesMessengerOfTheGods::Concerns::Message
+
+  attr_accessor :dude, :hrm
+
+  delf self.omg_dude(msg)
+    new(dude: msg["old_dude"], hrm: nil)
+  end
+end
+
+class AwesomeWorker < HermesMessengerOfTheGods::Worker
+  self.endpoint = {
+    default: sns_endpoint("sns::arn", jitter: false)
+  }
+  self.deserialize_with = MessageType
+  self.deserialize_method = :omg_dude
+end
+```
+
+If the endpoint emits a hash with only the keys `:dude` and `:hrm` you can get
+away without defining any `deserialize_with` setting. However if you need to do
+some translations, you can define the custom builder function such as `omg_dude`
+above.
+
+### Execution
+For a single queue `./bin/fly_hermes start --worker={worker job}`
+For pooled operation `./bin/fly_hermes start --pool {worker job}--{count} {worker2 job}--{count} {worker3 job}--{count}
+
+## Message & Worker in one class
+When your message meets a few constraints you can combine the logic of the worker
+and the message into a single class.
+
+* You may only use one endpoint (due to limitation on the worker)
+* The endpoint must be both readable and writable (No SNS for example)
+* You do not need to customize callbacks much
+
+```ruby
+class MonoMessageWorker
+  include HermesMessengerOftheGods::Concerns::MonoMessage
+
+  self.endpoints = { default: sqs_endpoint("arn") } # Only 1 endpoint, options are supported
+
+  def perform
+    # Do magic Here
+  end
+end
+```
+
+You may then start the message execution with:
+
+`./bin/fly_hermes start --pool=MonoMessageWorker--1`
+
+
+## Protocol Buffers
+Hermes supports using ProtoBuffers over the wire using the
+`HermesMessengerOfTheGods::Concerns::GrpcProtobuf` mixin. Currently only JSON
+encoding is allowed.
+
+Example:
+
+```ruby
+class MessageA
+  include HermesMessengerOfTheGods::Concerns::Message
+  include HermesMessengerOfTheGods::Concerns::GrpcProtobuf
+  self.protobuf_class = Helloworld::HelloRequest
+end
+
+# Or
+
+class MessageB
+  include HermesMessengerOftheGods::Concerns::MonoMessage
+  include HermesMessengerOfTheGods::Concerns::GrpcProtobuf
+  self.protobuf_class = Helloworld::HelloRequest
+end
+```
+
+This will use the Protobuf encoder and decoder for transmission on the wire.
+
+### Notes:
+
+1) Message initialization now requires the first parameter be an instance
+   of the provided protobuf.
+2) If you are are using Dispatch helpers, you probably need to define a
+   specific builder.
+3) If you are lazy, you can use `.from_message` and pass a hash.
+
+## Health Check HTTP Server
+Hermes comes with a built in HTTP Health check server that may be optionally enabled. The server responds with `200 OK`
+when everything is going okay, and `500 Internal Server Error` when things seem out of wack.
+
+The basic logic of the sever is as follows:
+
+1) The server becomes unhealthy if no work has been performed in 60 seconds (customizable with the environment
+   variable `HERMES_MINIMUM_WORK_FREQUENCY`) AND the endpoint reports there is work to do.
+2) The server always reports healthy when there is no pending work to be done.
+
+Pending work for an SQS queue is when there is at least one message visible in the queue.
+
+In order to enable the HTTP Health Check server, you should set the environment variable `ENABLE_HERMES_HEALTH_CHECK` to
+and value except `false`. The port the server listens on is configurable using `HERMES_WORKER_STATUS_PORT` defaulting to 4242.
+
+
+## Endpoints
+Endpoints provide the external communication layer to the outside world.
+Generally speaking all endpoints should inherit from the base endpoint to
+provide common behaviors.
+
+Endpoint creation takes two parameters, the first is the "endpoint", the second
+is a options hash. Endpoint is required, while options provide more fine
+grained control of the endpoint behavior. The content of both parameters is
+dependent on the backing endpoint.
+
+### General Endpoint Behavior
+Each end point needs to define a `transmit` method. Beyond that the basic
+behavior common to all end points will be automatically added.
+
+### Dispatching a message to an endpoint
+Dispatching a message into a endpoint can be accomplished by calling `dispatch`
+and passing the message in as the first parameter.
+
+* `dispatch(message)` returns true or false to represent success or failure
+* `dispatch!(message)` returns true on success, and raises the last error received on  failure
+
+### Message Transformations
+The endpoint will request the content to deliver from the message object itself.
+A heirachy of methods will be used to determine wich method to use.
+
+1. You may provide an options during creation of the endpoint, `:transformer` and that will be used. This can be a proc, or a method name. The proc will be called with the object as the first parameter
+2. `to_<endpoint_type>_message` - For any endpoint, the class name will be transformed into a dynamic method call. For example the SnsFoo endpoint will look for `to_sns_foo_message`.
+3. `to_message` - Generic to_message handler for the class.
+4. Finally the transmitted message itself. You probably don't want this since most message objects do not serialize well.
+
+### Backoffs
+By default a linear backoff is used of 1 second per number of tries. You can
+configure this by passing the `:backoff` option with `:linear` or
+`:exponential`. If you prefer to write your own backoff, you can also pass a
+proc to the backoff method. A `:backoff` value of nil will disable any backoffs.
+
+Jitter is added by default to the backoff time. This can be disabled by passing
+`jitter: false` in with the options.
+
+### Error Handling
+By default all standard errors are caught and retried, with the exceptoin of
+`HermesMessengerOfTheGods::Endpoints::FatalError`. If this error is raised
+there will be no further retries.
+
+All errors raised during execution will be stored in the `endpoint.errors`
+array.
+
+### SNS Endpoint
+* `endpoint` - the ARN of the SNS endpoint to publish to.
+* Avaialable options:
+  * `:client_options` - a hash of options to pass to the [SNS::Topic](http://docs.aws.amazon.com/sdkforruby/api/Aws/SNS/Topic.html) during creation. If your ENV is setup with all required AWS keys you won't need to set anything here.
+  * `:publish_options` - a hash of options to pass into the publish command for the [SNS::Topic](http://docs.aws.amazon.com/sdkforruby/api/Aws/SNS/Topic.html#publish-instance_method). The message key will always be overwritten. You can also provide a proc, which will be passed the message per transmission and is expected to return a hash with options. A proc can be used to set message_attributes that rely on variables from the message body.
+
+### SQS Endpoint
+* `endpoint` - the URL of the SQS queue to poll.
+* Avaialable options:
+  * `:client_options` - a hash of options to pass to the [SNS::Queue](http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/SQS/Queue.html) during creation. If your ENV is setup with all required AWS keys you won't need to set anything here.
+  * `:poll_options` - a hash of options to pass into the poll command for the [SQS::QueuePoller](http://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/QueuePoller.html#poll-instance_method).
+  * `:send_options` - a hash of options to pass into the send_message command for the [SQS::Queue](http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/SQS/Queue.html#send_message-instance_method). The message_body key will always be overwritten. You can also provide a proc, which will be passed the message per transmission and is expected to return a hash with options.
+  * `:jsonify` - converts the trasnmitted data to json prior to transmission
+  * `:from_sns` - reads and converts from JSON the serialized Message key
+
+## Instrumentation
+HMOTG Uses `ActiveSupport::Notifications` to provide internal instrumentation
+
+All messages emited include the emitter in their payload if possible:
+* Messages prefixed with `hermes_messenger_of_the_gods.worker` include the
+  `worker` key.
+* Messages prefixed with `hermes_messenger_of_the_gods.message` include the
+  `message` key.
+* Messages prefixed with `hermes_messenger_of_the_gods.endpoint` include the
+  `endpoint` key.
+
+**NOTE:** All messages are prefixed with `hermes_messenger_of_the_gods` but it
+was omitted here for brevity.
+
+| Message                     | Is Timed | Description                                                          | Payload Objects                                                                                                                                                                            |
+| ------                      | -------- | -----------                                                          | ---------------                                                                                                                                                                            |
+| `worker.starting`           |          | Called when the worker starts up                                     |                                                                                                                                                                                            |
+| `worker.starting_job`       |          | Called before work begins on a job                                   | `job` - the deserialize job being run                                                                                                                                                      |
+| `worker.run_job`            | ✓        | Measured the execution of the message                                | `job` - the deserialize job being run                                                                                                                                                      |
+| `worker.failure`            |          | Invoked when a message raises an error during execution              | `job` - the deserialize job being run <br> `error` - the exception raised                                                                                                                  |
+| `worker.fatal_error`        |          | Raised when an unhandled exception causes the worker to die          | `exception` - The exceptoin that caused the worker to die                                                                                                                                  |
+| `worker.deserialization`    | ✓        | Called during message deserialization off the endpoint               | `job` - the raw job received from the endpoint                                                                                                                                             |
+| `message.dispatch`          | ✓        | Called when a message complets the dispatch process                  |                                                                                                                                                                                            |
+| `message.dispatch_failure`  |          | Called once for every failed enpoint                                 | `exception` - the error raised <br> `endpoint_name` - the name of the failed endpoint                                                                                                      |
+| `endpoint.dispatch`         | ✓        | Called around the actual dispatch for each endpoint                  |                                                                                                                                                                                            |
+| `endpoint.dispatch_failure` |          | Called each time the endpoint fails to dispatch                      | `try` - The number of times the endpoint has tried to dispatch so far. First call is 1. <br> `exception` - The error that was raised causing the failure                                   |
+| `endpoint.final_failure`    |          | Called on the last exception to before the endpoint gives up         | `try` - The number of times the endpoint has tried to dispatch so far. At this point `try` equals the `:retries` config . <br> `exception` - The error that was raised causing the failure |
+| `endpoint.read_failure`     |          | An inbound message could not be handled (Malformed JSON for example) | `exception` - The unhandled error                                                                                                                                                          |
+
+
+## Global Configuration
+When configuring HMOTG you can set global configuration opions like so:
+
+```ruby
+HermesMessengerOfTheGods.config do |config|
+  config.config_options = wassup
+end
+```
+
+Allowed configuration options:
+
+| Configuration Option | Default               | Allowed Values
+| -------------------- | --------------------- | --------------
+| logger               | Ruby Logger to STDOUT | A single or array of logger like objects
+| quiet                | false                 | Set this to true and all logging will be disabled
+| delay_strategy       | :smart                | This value can be a symbol representing the delay strategy, or a proc which will be expected to do the delaying
+| delay_options        | {}                    | A set of default options to use when delaying. The contents of this hash are dependent on the `delay_strategy` used
+
+## Logging
+`HermesMessengerOfTheGods::Message`, `HermesMessengerOfTheGods::Worker` have
+access to a unified set of logging helpers which respect the global logging
+configuration. These helpers correspond to the configured `Logger::Severity`
+constant. `debug`, `info`, `warn`, `error`, and `fatal` are provided by default.
+
+All methods which correspond to a severity level take a single message
+parameter, or a block which is expected to return the message.
+[With the same behavior as the default logger](http://ruby-doc.org/stdlib-1.9.3/libdoc/logger/rdoc/Logger.html#class-Logger-label-How+to+log+a+message).
+
+
+Additionally the `say` method is provided which takes the desired level contsant
+as the first parameter. `say(Logger::ERROR, "foo")`
+
+### Outputters
+Hermes ships and attaches a Basic logger for by default. The primary warning is
+that the basic output will default to the `inspect` output of your message. If
+your message contains a significant number of, or complex, instance variables
+There may be more info logged then is necessary to identify the message.
+
+As such you can overwrite the `inspect` method, or create the custom
+`to_log_s` method which will be used instead.
+
+Example regular output:
+```
+I, [date stamp]  INFO -- Worker WorkerClass::0(pid: 8): Starting Job <MessageClass: {:instance_var_1 => 123, :instance_var_2=5678}>
+I, [date stamp]  INFO -- Worker WorkerClass::0(pid: 8): Finished Job <MessageClass: {:instance_var_1 => 123, :instance_var_2=5678}>
+```
+
+Don't want any output such as during a test suite, or want to write your own?
+You can turn off the Basic outputter via
+`HermesMessengerOfTheGods::Output::Basic.unsubscribe!`
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/hermes-messenger-of-the-gods.
+