rocketman 0.1.1 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7add76c53ecb1c5996cc2c21e0141bcadeb4169d40bb9785c577286839eff7a7
-  data.tar.gz: e2b0ff4413172768ef6a331b76fa75113fe80cbb1aa63a9fc8e62bba915b4b8e
+  metadata.gz: 52b5a1cb157137e4894f21237379ea6bfa577c1029cf3f6cfc435eb18b19c767
+  data.tar.gz: 1d47bc328a3ff19de8c15acd34ada270343d626335406878c967b4b734e3f213
 SHA512:
-  metadata.gz: da1cd6a839cb80ae22eebb8ca1150bfacd555999a4ada7c348838647db7118d5e978641023459dce4c8ef2158d70043d84185d2e876153e2cea444b69616f266
-  data.tar.gz: 37b70df7e25868cfe7d3de36b38d49c116e75a699b7a6ee680491ab4c428f212f24dab728f570e1266228483daed1074d15876a37432a8a22c2ef0ef80939ee6
+  metadata.gz: 26f006bd824c1993463984c6b57401fb866ad2293585c72b01506000e6af18f9358d90cd9b8558f17796820f7d20d4aa0da145cdcdd490e358d396788d187a32
+  data.tar.gz: 05ee47a76e61a3d6700e28a1208d80e856731d160330c373fa86cd4d38a471b8f7bcba481d9a55924dcc5ddd7171ccc311a355426919ec3763c2ff33cd18fd2f

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    rocketman (0.1.1)
+    rocketman (0.2.0)
 
 GEM
   remote: https://rubygems.org/
@@ -13,6 +13,7 @@ GEM
       coderay (~> 1.1.0)
       method_source (~> 0.9.0)
     rake (10.5.0)
+    redis (4.1.2)
     rspec (3.8.0)
       rspec-core (~> 3.8.0)
       rspec-expectations (~> 3.8.0)
@@ -34,6 +35,7 @@ DEPENDENCIES
   bundler (~> 1.17)
   pry
   rake (~> 10.0)
+  redis
   rocketman!
   rspec (~> 3.0)
 

data/README.md

@@ -3,7 +3,9 @@
 <sub>*yes, I know it says Starman on the image*</sub>
 > *🎶 And I think it's gonna be a long long time 'Till touch down brings me round again to find 🎶*
 
-Rocketman is a gem that introduces Pub-Sub mechanism within Ruby code.
+Rocketman is a gem that introduces Pub-Sub mechanism within your Ruby code.
+
+The main goal of Rocketman is not to replace proper message buses like Redis PubSub/Kafka, but rather be a stepping stone. You can read more about the [rationale behind the project](https://github.com/edisonywh/rocketman#why-use-rocketman-rather-than-a-proper-message-bus-eg-redis-pubsubkafka) down below.
 
 As with all Pub-Sub mechanism, this greatly decouples your upstream producer and downstream consumer, allowing for scalability, and easier refactor when you decide to move Pub-Sub to a separate service.
 
@@ -29,20 +31,84 @@ Or install it yourself as:
 
 Rocketman exposes two module, `Rocketman::Producer` and `Rocketman::Consumer`. They do exactly as what their name implies. All you need to do is `include Rocketman::Producer` and `extend Rocketman::Consumer` into your code.
 
-#### Producer
+### Producer
 Producer exposes one **instance** method to you: `:emit`. `:emit` takes in the event name and an optional payload and publishes it to the consumers. There's nothing more you need to do. The producer do not have to know who its consumers are.
 
 ```ruby
 class Producer
-    include Rocketman::Producer
+  include Rocketman::Producer
+
+  def hello_world
+    emit :hello, payload: {"one" => 1, "two" => 2}
+  end
+end
+```
+
+Note that Producer emit events with threads that run in a thread pool. The default number of worker is 5, and the workers default to checking the job with a 3 seconds interval. You can tweak these to your liking, refer to the [`Configuration` section](https://github.com/edisonywh/rocketman#configuration) below for more informations.
+
+### Consumer
+Consumer exposes a **class** method, `:on_event`. `:on_event` takes in the event name, and also an additional block, which gets executed whenever a message is received. If an additional `payload` is emitted along with the event, you can get access to it in the form of block argument.
+
+```ruby
+class Consumer
+  extend Rocketman::Consumer
 
-    def hello_world
-        emit :hello, payload: {"one" => 1, "two" => 2}
-    end
+  on_event :hello do |payload|
+    puts "I've received #{payload} here!"
+    # => I've received {:payload=>{"one"=>1, "two"=>2}} here!
+  end
 end
 ```
 
-Note that Producer emit events with threads that run in a thread pool. The default number of worker is 5, and the workers default to checking the job with a 3 seconds interval. You can tweak it to your liking like so:
+Simple isn't it?
+
+#### Consume events from external services
+
+If you want to also consume events from external services, you're in luck (well, as long as you're using `Redis` anyway..)
+
+Rocketman exposes a `Rocketman::Bridge`, which allows your Ruby code to start consuming events from Redis, **without any changes to your consumers**.
+
+This works because `Bridge` will listen for events from those services on behalf of you, and then it'll push those events onto the internal `Registry`.
+
+**This pattern is powerful because this means your consumers do not have to know where the events are coming from, as long as they're registed onto `Registry`.**
+
+Right now, only `Redis` is supported. Assuming you have the `redis` gem installed, this is how you register a bridge.
+
+```ruby
+Rocketman::Bridge.construct(Redis.new)
+```
+
+That's all! Rocketman will translate the following
+
+```
+redis-cli> PUBLISH hello payload
+```
+
+to something understandable by your consumer, so a consumer only has to do:
+
+```ruby
+on_event :hello do |payload|
+  puts payload
+end
+```
+
+Notice how it behaves exactly the same as if the events did not come from Redis :)
+
+**NOTE**: You should always pass in a **new, dedicated** connection to `Redis` to `Bridge#construct`. This is because `redis.subscribe` will hog the whole Redis connection (not just Ruby process), so `Bridge` expects a dedicated connection for itself.
+
+## Persisting emitted events
+
+By default, the events emitted from your app will be stored in an in-memory `Queue`, which will get processed by Rocketman threaded workers.
+
+However this also means that if your app dies with events still in your job queue, your emitted events which are stored in-memory will be lost.
+
+That is obviously not desirable, so that's why **Rocketman ships with an option to use `Redis` as your backing storage mechanism.**
+
+All you need to do is pass in a `Redis` connection to Rocketman. Refer to the [`Configuration` section below](https://github.com/edisonywh/rocketman#configuration) for more information.
+
+## Configuration
+
+Here are the available options to tweak for Rocketman.
 
 ```ruby
 # config/initializers/rocketman.rb
@@ -50,32 +116,32 @@ Note that Producer emit events with threads that run in a thread pool. The defau
 Rocketman.configure do |config|
   config.worker_count = 10 # defaults to 5
   config.latency      = 1  # defaults to 3, unit is :seconds
+  config.storage      = Redis.new # defaults to `nil`
+  config.debug        = true # defaults to `false`
 end
 ```
 
-#### Consumer
-Consumer exposes a **class** method, `:on_event`. `:on_event` takes in the event name, and also an additional block, which gets executed whenever a message is received. If an additional `payload` is emitted along with the event, you can get access to it in the form of block argument.
+Currently `storage` only supports `Redis`, suggestions for alternative backing mechanisms are welcomed.
 
-```ruby
-class Consumer
-    extend Rocketman::Consumer
+`debug` mode enables some debugging `puts` statements, and also tweak the `Thread` workers to `abort_on_exception = true`. So if you have failing jobs, this is how you can figure out what's happening inside your workers.
 
-    on_event :hello do |payload|
-        puts "I've received #{payload} here!"
-        # => I've received {:payload=>{"one"=>1, "two"=>2}} here!
-    end
-end
-```
+## Why use `Rocketman`, rather than a proper message bus (e.g Redis PubSub/Kafka)?
 
-Simple isn't it?
+It is worth noting that `Rocketman` is not meant to be a replacement for the aforementioned projects -- both Redis PubSub and Kafka are battle-tested and I highly encourage to use them if you can.
+
+**But**, `Rocketman` recognizes that it's not an easy task to spin up an external message bus to support event-driven architecture, and that's what it's trying to do - to be a stepping stone for eventual greatness.
+
+Moving onto a event-driven architecture is not an easy task - your team has to agree on a message bus, the DevOps team needs the capacity to manage the message bus, and then what about clustering? failovers?
+
+So what Rocketman offers you is that you can start writing your dream-state event-driven code **today**, and when the time comes and your team has the capacity to move to a different message bus, then it should be a minimal change.
 
 ## Roadmap
 
 Right now events are using a `fire-and-forget` mechanism, which is designed to not cause issue to producers. However, this also means that if a consumer fail to consume an event, it'll be lost forever. **Next thing on the roadmap is look into a retry strategy + persistence mechanism.**
 
-Events are also stored in memory in `Rocketman::Registry`, which has the same problem as above. **Something to think about is to perhaps move it onto a persistent storage, like Redis for example.**
+~~Emitted events are also stored in memory in `Rocketman::Pool`, which means that there's a chance that you'll lose all emitted jobs. Something to think about is to perhaps move the emitted events/job queue onto a persistent storage, like Redis for example.~~ **Redis support is now available!**
 
-The interface could also probably be better defined, as one of the goal of Rocketman is to be the stepping stone before migrating off to a real, proper message queue/pub-sub mechanism like Kafka. **I want to revisit and think about how can we make that transition more seamless?**
+The interface could also probably be better defined, as one of the goal of Rocketman is to be the stepping stone before migrating off to a real, proper message queue/pub-sub mechanism like Kafka. **I want to revisit and think about how can we make that transition more seamless.**
 
 ## Development
 

data/lib/rocketman.rb

@@ -4,3 +4,4 @@ require 'rocketman/registry.rb'
 require 'rocketman/event.rb'
 require 'rocketman/producer.rb'
 require 'rocketman/consumer.rb'
+require 'rocketman/bridge.rb'

data/lib/rocketman/bridge.rb

@@ -0,0 +1,29 @@
+module Rocketman
+  class Bridge
+    include Rocketman::Producer
+
+    attr_reader :service
+
+    def initialize(service)
+      @service = service
+    end
+
+    def self.construct(service)
+      instance = new(service)
+
+      case instance.service.class.to_s
+      when "Redis"
+        puts "Rocketman> Using Redis as external producer".freeze
+        Thread.new do
+          instance.service.psubscribe("*") do |on|
+            on.pmessage do |_pattern, event, payload|
+              instance.emit(event, payload)
+            end
+          end
+        end
+      else
+        puts "Rocketman> Don't know how to handle service: `#{service}`"
+      end
+    end
+  end
+end

data/lib/rocketman/config.rb

@@ -8,11 +8,13 @@ module Rocketman
   end
 
   class Configuration
-    attr_accessor :worker_count, :latency
+    attr_accessor :worker_count, :latency, :storage, :debug
 
     def initialize
       @worker_count = 5
       @latency = 3
+      @storage= nil
+      @debug = false
     end
   end
 end

data/lib/rocketman/consumer.rb

@@ -2,14 +2,8 @@ module Rocketman
   module Consumer
     def on_event(event, &action)
       consumer = self
-      Rocketman::Registry.instance.register_event(event)
-      register_consumer(event, consumer, action)
-    end
-
-    private
-
-    def register_consumer(event, consumer, action)
-      Rocketman::Registry.instance.register_consumer(event, consumer, action)
+      Rocketman::Registry.register_event(event)
+      Rocketman::Registry.register_consumer(event, consumer, action)
     end
   end
 end

data/lib/rocketman/event.rb

@@ -3,12 +3,13 @@ module Rocketman
     def initialize(event, payload)
       @event = event
       @payload = payload
-      @test = payload.fetch(:test, false)
-      Rocketman::Registry.instance.register_event(event)
+      Rocketman::Registry.register_event(event)
     end
 
     def notify_consumers
-      consumers = Rocketman::Registry.instance.get_consumers_for(@event)
+      consumers = Rocketman::Registry.get_consumers_for(@event)
+
+      return if consumers.nil? || consumers.empty?
 
       consumers.each do |consumer, action|
         consumer.instance_exec(@payload, &action)

data/lib/rocketman/job_queue.rb

@@ -0,0 +1,76 @@
+require 'forwardable'
+require 'json'
+
+module Rocketman
+  class JobQueue
+    extend Forwardable
+
+    QUEUE_KEY = "rocketman".freeze
+
+    def_delegators :@jobs, :<<, :empty?, :size, :clear, :push, :pop
+
+    def initialize
+      @storage = Rocketman.configuration.storage
+      @jobs = get_job_queue
+
+      at_exit { persist_events } if @storage.class.to_s == "Redis"
+    end
+
+    def schedule(job)
+      @jobs << job
+    end
+
+    private
+
+    def get_job_queue
+      case @storage.class.to_s
+      when "Redis"
+        rehydrate_events
+      else
+        Queue.new
+      end
+    end
+
+    def rehydrate_events
+      queue = Queue.new
+
+      if raw_data = @storage.get(QUEUE_KEY)
+        puts "Rehydrating Rocketman events from #{@storage.class}" if Rocketman.configuration.debug
+
+        rehydrate = JSON.restore(raw_data) # For security measure to prevent remote code execution (only allow contents valid in JSON)
+        jobs = Marshal.load(rehydrate)
+        event_count = 0
+
+        until jobs.empty?
+          queue << jobs.shift
+          event_count += 1
+        end
+
+        puts "Rehydrated #{event_count} events from #{@storage.class}" if Rocketman.configuration.debug
+
+        @storage.del(QUEUE_KEY) # After rehydration, delete it off Redis
+      end
+
+      queue
+    end
+
+    def persist_events
+      return if @jobs.empty?
+
+      puts "Persisting Rocketman events to #{@storage.class}" if Rocketman.configuration.debug
+      intermediary = []
+      event_count = 0
+
+      until @jobs.empty?
+        intermediary << @jobs.pop
+        event_count += 1
+      end
+      @jobs.close
+
+      marshalled_json = Marshal.dump(intermediary).to_json # For security measure to prevent remote code execution (only allow contents valid in JSON)
+
+      @storage.set(QUEUE_KEY, marshalled_json)
+      puts "Persisted #{event_count} events to #{@storage.class}" if Rocketman.configuration.debug
+    end
+  end
+end

data/lib/rocketman/pool.rb

@@ -1,15 +1,18 @@
 require 'singleton'
+require 'rocketman/job_queue'
 
 module Rocketman
   class Pool
     include Singleton
 
+    attr_reader :jobs
+
     def initialize
       worker_count = Rocketman.configuration.worker_count
       latency = Rocketman.configuration.latency
 
       @latency = latency
-      @jobs = Queue.new
+      @jobs = Rocketman::JobQueue.new
       @workers = []
 
       worker_count.times do
@@ -19,17 +22,15 @@ module Rocketman
       # spawn_supervisor # TODO: Write a supervisor to monitor workers health, and restart if necessary
     end
 
-    def schedule(&job)
-      @jobs << job
-    end
-
     private
 
     def spawn_worker
+      Thread.abort_on_exception = true if Rocketman.configuration.debug
+
       Thread.new do
         loop do
           job = @jobs.pop
-          job.call
+          job.notify_consumers # Job is an instance of Rocketman::Event
           sleep @latency
         end
       end

data/lib/rocketman/producer.rb

@@ -1,8 +1,8 @@
 module Rocketman
   module Producer
-    def emit(event, **payload)
-      event = Rocketman::Event.new(event, payload)
-      Rocketman::Pool.instance.schedule { event.notify_consumers }
+    def emit(event, payload = {})
+      event = Rocketman::Event.new(event.to_sym, payload)
+      Rocketman::Pool.instance.jobs.schedule(event)
     end
   end
 end

data/lib/rocketman/registry.rb

@@ -1,4 +1,5 @@
 require 'singleton'
+require 'forwardable'
 
 module Rocketman
   class Registry
@@ -20,6 +21,10 @@ module Rocketman
       @registry[event][consumer] = action
     end
 
+    def get_events
+      @registry.keys
+    end
+
     def get_consumers_for(event)
       @registry[event]
     end
@@ -27,5 +32,11 @@ module Rocketman
     def event_exists?(event)
       !@registry[event].nil?
     end
+
+    # This is to help hide the Singleton interface from the rest of the code
+    class << self
+      extend Forwardable
+      def_delegators :instance, *Rocketman::Registry.instance_methods(false)
+    end
   end
 end

data/lib/rocketman/version.rb

@@ -1,3 +1,3 @@
 module Rocketman
-  VERSION = "0.1.1"
+  VERSION = "0.2.0"
 end

data/rocketman.gemspec

@@ -23,6 +23,7 @@ Gem::Specification.new do |spec|
   spec.require_paths = ["lib"]
 
   spec.add_development_dependency "pry"
+  spec.add_development_dependency "redis"
   spec.add_development_dependency "bundler", "~> 1.17"
   spec.add_development_dependency "rake", "~> 10.0"
   spec.add_development_dependency "rspec", "~> 3.0"

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: rocketman
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.2.0
 platform: ruby
 authors:
 - Edison Yap
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2019-08-11 00:00:00.000000000 Z
+date: 2019-08-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: pry
@@ -24,6 +24,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: redis
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
@@ -85,9 +99,11 @@ files:
 - bin/console
 - bin/setup
 - lib/rocketman.rb
+- lib/rocketman/bridge.rb
 - lib/rocketman/config.rb
 - lib/rocketman/consumer.rb
 - lib/rocketman/event.rb
+- lib/rocketman/job_queue.rb
 - lib/rocketman/pool.rb
 - lib/rocketman/producer.rb
 - lib/rocketman/registry.rb