message_bus_client_worker 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5673b34ade6dd5f7bf7ce503fb76e3167f421d260c3d5499b7865996b975d042
-  data.tar.gz: 728c1b77a4471e04b99c9df3d2da26f6b07413d476ed5a0351a21b76bc5b6902
+  metadata.gz: de30a005fd8d0ded3dd3abb9539f67efc2e06b526212af01a37b7c0fd1d43d31
+  data.tar.gz: 73de613d059e2d77b5732eeac9c6581fa25de7cdd6c1d686864059d700916238
 SHA512:
-  metadata.gz: 80cebcc4826049b0d7261b1b554a44657e5b61375f342691390d2ba7c20f5f5c4ae8ad34a2a564821115b4bb66c65891f6ca560ce34ec922f3a548a264607725
-  data.tar.gz: 6888b78df0594aae1db838ba222983637e14e478e18c40078a5c2464e3ea46d05cd440cecb870e092a605c16008753c8b9ffc0cacb8284f98048cd04ed8a2f23
+  metadata.gz: 10a046ecaa85f5e5d92ce11b50d6cdbeebc152ba684d9514326c858dee857009186640161834f3457244cf3972d5d0e78b51f20012d868a30797fc4a7211d555
+  data.tar.gz: 47b26efdcc4869834fba1e49f5a44c0e8336551426c2708e2fbd101b4b3de95e145aac079d6ffd6ccb426b65140cd7f3aa05082b3edbf6f3ba987aa2aed5d7cc

data/CHANGELOG.md

@@ -4,6 +4,10 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+## [0.2.0] - 2018-08-08
+### Changed
+- Ability to specify what message_id to start from
+
 ## [0.1.0] - 2018-08-07
 ### Added
 - Initial release

data/Gemfile.lock

@@ -1,12 +1,14 @@
 PATH
   remote: .
   specs:
-    message_bus_client_worker (0.1.0)
+    message_bus_client_worker (0.2.0)
+      activesupport
       addressable
       gem_config
       http
       light-service
       sidekiq
+      sidekiq-unique-jobs (>= 6.0.0)
 
 GEM
   remote: https://rubygems.org/
@@ -74,6 +76,11 @@ GEM
       connection_pool (~> 2.2, >= 2.2.0)
       rack-protection (>= 1.5.0)
       redis (>= 3.3.5, < 5)
+    sidekiq-unique-jobs (6.0.4)
+      concurrent-ruby (~> 1.0, >= 1.0.5)
+      sidekiq (>= 4.0, < 6.0)
+      thor (~> 0)
+    thor (0.20.0)
     thread_safe (0.3.6)
     tzinfo (1.2.5)
       thread_safe (~> 0.1)
@@ -86,7 +93,6 @@ PLATFORMS
   ruby
 
 DEPENDENCIES
-  activesupport
   bundler (~> 1.16)
   message_bus_client_worker!
   pry-byebug

data/README.md

@@ -2,7 +2,7 @@
 
 Subscribe to [MessageBus](https://github.com/SamSaffron/message_bus) using Sidekiq workers. This gem was borne out of the noisy logs and difficult in debugging using [message_bus-client](https://github.com/lowjoel/message_bus-client). This gem aims to:
 
-- allow sane debugging by having thread-related code all over the calls to the MessageBus channel by relying on Sidekiq
+- allow sane debugging. Do not have thread-related code all over the code to the MessageBus channel. Rely on Sidekiq
 - keep subscriptions to 1 per channel. With current options, every web server process would start a thread that listened to the MessageBus channels.
 - do not unnecessarily add noise when starting the console like `rails console`
 - recover from downtime by keeping track of the last message it processed per channel
@@ -36,25 +36,67 @@ MessageBusClientWorker.configure do |c|
     #   "/b_channel" => "BChannelPayloadProcessor",
     # }
     "https://etc.com" => {
-      "/exchange_rates" => "ProcessExchangeRate",
-      "/messages" => "ProcessMessage",
+      "/exchange_rates" => {
+        processor: "ProcessExchangeRate",
+        message_id: 0,
+      },
+      "/messages" => { processor: "ProcessMessage" },
     },
     "https://someotherdomain.com" => {
-      "/errors" => "ProcessError",
+      "/errors" => { processor: "ProcessError" },
     },
   }
 end
 ```
 
+### Processor
+
+The processor should look like this:
+
+```ruby
+class ProcessMessage
+  def self.call(data, payload)
+    # ...
+  end
+end
+```
+
+http://chat.samsaffron.com's `/message` channel returns JSON like this:
+
+```json
+[
+  {"global_id":1478,"message_id":3,"channel":"/message","data":{"data":"hey","name":"joe"}},
+  {"global_id":1479,"message_id":4,"channel":"/message","data":{"data":"what's up","name":"joe"}},
+  ...
+]
+```
+
+The processor you define will receive `call` for every element in the JSON. In `ProcessMessage` processor as seen above:
+
+- `data` would be the value of the "data" key as a Ruby hash, i.e. `{"data" => "hey","name" => "joe"}`
+- `payload` is the whole item as a Ruby hash, i.e. `{"global_id"=>1479, "message_id"=>4, "channel"=>"/message", "data"=>{"data"=>"what's up", "name"=>"joe"}}`
+
+If you don't care to see the whole payload, you can do the following:
+
 ```ruby
 class ProcessMessage
-  def self.call(payload)
-    payload # {"global_id":1478,"message_id":3,"channel":"/message","data":{"data":"hey","name":"joe"}}
+  def self.call(data, _)
+    # ...
   end
 end
 ```
 
-To keep the subscription alive if the worker dies or during restarts, use a gem like [sidekiq-cron](https://github.com/ondrejbartas/sidekiq-cron) and enqueue the `MessageBusClientWorker::EnqueuingWorker`:
+### `message_id`
+
+`message_id` defines where the worker should start reading from.
+
+- you do not want to use "-1" if you're short polling because no messages will be read and MessageBusClientWorker will not record the last `message_id` seen.
+- this is only used when nothing has been read before. If you put 0, but the last message that MessageBusClientWorker pulled had a `message_id` of `23`, then this gem will continue where it left off, and not read from 0
+- defaults to "-1"
+
+### Staying subscribed
+
+To keep the subscription alive if the worker dies, the app restarts, or the worker falls back to short-polling or long-polling without streaming, use a gem like [sidekiq-cron](https://github.com/ondrejbartas/sidekiq-cron) and enqueue the `MessageBusClientWorker::EnqueuingWorker`:
 
 ```yml
 message_bus_client_worker:
@@ -74,8 +116,8 @@ Every time `MessageBusClientWorker::EnqueuingWorker` is enqueued, `EnqueuingWork
 
 `SubscriptionWorker` will open a connection to the server, and try the following (not all have been implemented):
 
-- [ ] long-poll with streaming, or if streaming is not supported...
-- [ ] long-poll, or if long-polling is not supported...
+- [ ] long-poll with streaming, or if streaming is not supported by the server...
+- [ ] long-poll, or if long-polling is not supported by the server...
 - [x] short-poll
 
 ## Development
@@ -84,8 +126,7 @@ Copy the config and edit (if necessary)
 
 ```sh
 cp spec/config.yml{.sample,}
-docker-compose up sidekiq chat
-rspec spec
+docker-compose run gem bundle exec rspec spec
 ```
 
 ## Contributing

data/lib/message_bus_client_worker/services/poll.rb

@@ -12,8 +12,10 @@ module MessageBusClientWorker
         Polling::GenerateClientId,
         Polling::GenerateURI,
         Polling::GenerateParams,
-        Polling::GetMessages,
-        Polling::ProcessMessages,
+        Polling::GetPayloads,
+        iterate(:payloads, [
+          Polling::ProcessPayload,
+        ])
       )
     end
 

data/lib/message_bus_client_worker/services/polling/generate_params.rb

@@ -6,10 +6,15 @@ module MessageBusClientWorker
       expects :host, :subscriptions
       promises :params, :form_params
 
+      DEFAULT_MESSAGE_ID = "-1".freeze
+
       executed do |c|
         c.params = { dlp: 't' }
         c.form_params = c.subscriptions.each_with_object({}) do |sub, hash|
-          hash[sub[0]] = GetLastId.(c.host, sub[0])
+          custom_message_id = sub[1][:message_id] ? sub[1][:message_id].to_s : nil
+          hash[sub[0]] = GetLastId.(c.host, sub[0]) ||
+            custom_message_id ||
+            DEFAULT_MESSAGE_ID
         end
       end
 

data/lib/message_bus_client_worker/services/polling/get_last_id.rb

@@ -4,12 +4,7 @@ module MessageBusClientWorker
 
       def self.call(host, channel)
         hash_key = GenLastIdKey.(host, channel)
-
-        id = Sidekiq.redis do |r|
-          r.hget(SetLastId::CHANNEL_INDICES_NAME, hash_key)
-        end
-
-        id || "0"
+        Sidekiq.redis { |r| r.hget(SetLastId::CHANNEL_INDICES_NAME, hash_key) }
       end
 
     end

data/lib/message_bus_client_worker/services/polling/{get_messages.rb → get_payloads.rb}

@@ -1,14 +1,14 @@
 module MessageBusClientWorker
   module Polling
-    class GetMessages
+    class GetPayloads
       extend LightService::Action
 
       expects :params, :form_params, :uri
-      promises :messages
+      promises :payloads
 
       executed do |c|
         body = HTTP.post(c.uri, params: c.params, form: c.form_params).body
-        c.messages = JSON.parse(body.to_s)
+        c.payloads = JSON.parse(body.to_s)
       end
     end
   end

data/lib/message_bus_client_worker/services/polling/process_payload.rb

@@ -0,0 +1,23 @@
+module MessageBusClientWorker
+  module Polling
+    class ProcessPayload
+      extend LightService::Action
+
+      expects :host, :subscriptions, :payload
+
+      executed do |c|
+        payload = c.payload
+        channel = payload["channel"]
+        channel_config = c.subscriptions[channel]
+
+        next c if channel_config.nil?
+
+        processor_class = Kernel.const_get(channel_config[:processor])
+
+        SetLastId.(c.host, channel, payload["message_id"])
+        processor_class.(payload["data"], payload)
+      end
+
+    end
+  end
+end

data/lib/message_bus_client_worker/version.rb

@@ -1,3 +1,3 @@
 module MessageBusClientWorker
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

data/lib/message_bus_client_worker/workers/subscription_worker.rb

@@ -2,10 +2,10 @@ module MessageBusClientWorker
   class SubscriptionWorker
 
     include Sidekiq::Worker
-    sidekiq_options retry: false
+    sidekiq_options retry: false, lock: :until_executed
 
     def perform(host, subscriptions, long=false)
-      Poll.(host, subscriptions, long)
+      Poll.(host, subscriptions.with_indifferent_access, long)
     end
 
   end

data/lib/message_bus_client_worker.rb

@@ -4,6 +4,9 @@ require "http"
 require "light-service"
 require "securerandom"
 require "sidekiq"
+require "sidekiq-unique-jobs"
+require "active_support/core_ext/hash/indifferent_access"
+require "active_support/core_ext/string/inflections"
 require "message_bus_client_worker/version"
 require "message_bus_client_worker/workers/enqueuing_worker"
 require "message_bus_client_worker/workers/subscription_worker"
@@ -13,8 +16,8 @@ require "message_bus_client_worker/services/polling/set_last_id"
 require "message_bus_client_worker/services/polling/generate_client_id"
 require "message_bus_client_worker/services/polling/generate_uri"
 require "message_bus_client_worker/services/polling/generate_params"
-require "message_bus_client_worker/services/polling/get_messages"
-require "message_bus_client_worker/services/polling/process_messages"
+require "message_bus_client_worker/services/polling/get_payloads"
+require "message_bus_client_worker/services/polling/process_payload"
 require "message_bus_client_worker/services/poll"
 
 module MessageBusClientWorker

data/message_bus_client_worker.gemspec

@@ -35,11 +35,12 @@ Gem::Specification.new do |spec|
   spec.add_dependency "http"
   spec.add_dependency "addressable"
   spec.add_dependency "light-service"
+  spec.add_dependency "sidekiq-unique-jobs", ">= 6.0.0"
+  spec.add_dependency "activesupport"
 
   spec.add_development_dependency "bundler", "~> 1.16"
   spec.add_development_dependency "rake", "~> 10.0"
   spec.add_development_dependency "rspec", "~> 3.0"
   spec.add_development_dependency "rspec-sidekiq"
-  spec.add_development_dependency "activesupport"
   spec.add_development_dependency "wait"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: message_bus_client_worker
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Ramon Tayag
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2018-08-07 00:00:00.000000000 Z
+date: 2018-08-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: gem_config
@@ -80,6 +80,34 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: sidekiq-unique-jobs
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 6.0.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 6.0.0
+- !ruby/object:Gem::Dependency
+  name: activesupport
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
@@ -136,20 +164,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-- !ruby/object:Gem::Dependency
-  name: activesupport
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
 - !ruby/object:Gem::Dependency
   name: wait
   requirement: !ruby/object:Gem::Requirement
@@ -193,8 +207,8 @@ files:
 - lib/message_bus_client_worker/services/polling/generate_params.rb
 - lib/message_bus_client_worker/services/polling/generate_uri.rb
 - lib/message_bus_client_worker/services/polling/get_last_id.rb
-- lib/message_bus_client_worker/services/polling/get_messages.rb
-- lib/message_bus_client_worker/services/polling/process_messages.rb
+- lib/message_bus_client_worker/services/polling/get_payloads.rb
+- lib/message_bus_client_worker/services/polling/process_payload.rb
 - lib/message_bus_client_worker/services/polling/set_last_id.rb
 - lib/message_bus_client_worker/version.rb
 - lib/message_bus_client_worker/workers/enqueuing_worker.rb

data/lib/message_bus_client_worker/services/polling/process_messages.rb

@@ -1,20 +0,0 @@
-module MessageBusClientWorker
-  module Polling
-    class ProcessMessages
-      extend LightService::Action
-
-      expects :host, :subscriptions, :messages
-
-      executed do |c|
-        c.messages.each do |message|
-          channel = message["channel"]
-          processor_class = Kernel.const_get(c.subscriptions[channel])
-
-          SetLastId.(c.host, channel, message["message_id"])
-          processor_class.(message["data"])
-        end
-      end
-
-    end
-  end
-end