hutch-schedule 0.7.0 → 0.7.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 67a8a0f9f0a464729740d0bb57ae9d1d10d5005927de1e746fc440ea9c15bbc0
-  data.tar.gz: 5b66aca0219764dfefc211cf4ee55bdd17dc13d9cb02a61e5cce39fbb03ee470
+  metadata.gz: 51507037110c8e884ac519fe99c27f8547da70f5ada9744bc034a7ec10da5ee3
+  data.tar.gz: 45f2fcb4e686f86a13bcded021c661b69227c8eca5f5b0b96764a9ac840a8372
 SHA512:
-  metadata.gz: 4b2db45314d63e5ed2b2a12810146e1d09aa85194a2a575246d83fc4cd850de06f055dcff4e489bcf1889058fbd1891e0479c53a796efb8e7cfa3eb577dcd5cb
-  data.tar.gz: ac1bd4cd5e6469d1d4d80690c29119a5929f947669753f54504a619b69a93f07549147198fcb669f3315703ce6e18646e6a8267eead5415abee9f14c9150fab3
+  metadata.gz: 1c507eead9eef80897516c1296cf3eaa3df4674fbba1b188138ce06317a332cfe886291e596a403f5fa3573079f4b9650a0206252e5567b451bda54dadeb07e4
+  data.tar.gz: 2f38719bc7da6b3ea9a7826a5e82555336b81b46acb62622af6d33f7ab1f52679af6f77d7e048274168dcb1166a10eb5ead8fe29fe1ffa5200a4022022444e18

data/README.md

@@ -1,9 +1,26 @@
-# Hutch::Schedule
+# Hutch Schedule
 
 Add the schedule message function to [Hutch](https://github.com/gocardless/hutch).
 
 See [hutch-schedule-demo](https://github.com/wppurking/hutch-schedule-demo) how to integration with rails.
 
+## Contents
+
+- [Hutch Schedule](#hutch-schedule)
+  - [Contents](#contents)
+  - [Installation](#installation)
+  - [Usage](#usage)
+  - [Hutch::Enqueue](#hutchenqueue)
+  - [Hutch::Threshold](#hutchthreshold)
+  - [Error Retry](#error-retry)
+  - [Rails](#rails)
+    - [Work with Hutch it`s self](#work-with-hutch-its-self)
+    - [Work with ActiveJob](#work-with-activejob)
+  - [Limits](#limits)
+  - [Contributing](#contributing)
+  - [License](#license)
+  - [TODO](#todo)
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -36,7 +53,17 @@ They will do something below:
   - Set `x-message-ttl: <30.days>`: to avoid the queue is to large, because there is no consumer with this queue.
 3. If ActiveJob is loaded. it will use `ActiveJob::Base.descendants` to register all ActiveJob class to one-job-per-consumer to Hutch::Consumer 
 
-### Hutch::Enqueue
+## Configurations
+
+Name | Default Value | Description
+-----|---------------|-------------
+worker_pool_size | 20 | Monkey patch the `Hutch::Worker` set the FixedThreadPool thread size(not the bunney ConsumerWorkPool size) 
+poller_interval| 1 | seconds of the poller to trigger, poller the message in BufferQueue submit to FixedThreadPool
+poller_batch_size | 100 | the message size of every batch triggerd by the poller
+redis_url | redis://127.0.0.1:6379/0 | Redis backend url for Ratelimit and Unique Job 
+ratelimit_bucket_interval | 1 | Ratelimit use the time bucket (seconds) to store the counts, lower the more accurate
+
+## Hutch::Enqueue
 Let consumer to include `Hutch::Enqueue` then it has the ability of publishing message to RabbitMQ with the `consume '<routing_key>'`
 
 * enqueue: just publish one message
@@ -53,7 +80,42 @@ We design the fixed delay level from seconds to hours, below is the details:
 RabbitMQ is not fit for storage lot`s of delay message so if you want delay an message beyand 3 hours so you need to storage it
 into database or some place.
 
-### Error Retry
+## Hutch::Threshold
+Let consumer to include `Hutch::Threshold` to get the ability of threshold the message consume. It's automatic included when 
+consumer include `Hutch::Enqueue`.
+
+1. static configuration
+```ruby
+class PlanConsumer
+  include Hutch::Consumer
+  include Hutch::Enqueue
+  
+  attempts 3
+  consume 'abc.plan'
+  # threshold 3 jobs per second
+  threshold rate: 3, interval: 1
+end
+```
+
+2. dynamic lambada
+```ruby
+class PlanConsumer
+  include Hutch::Consumer
+  include Hutch::Enqueue
+  
+  attempts 3
+  consume 'abc.plan'
+  # threshold 2 jobs every 2 second with get_report context
+  threshold -> { { context: 'get_report', rate: 2, interval: 2 } }
+end
+```
+
+threshold lambada need get return value must be a Hash and include:
+* context: the limit context with currency threshold
+* rate: the rate speed of threshold
+* interval: the time range of threshold
+
+## Error Retry
 If you want use error retry, then:
 
 1. Add `Hutch::ErrorHandlers::MaxRetry` to `Hutch::Config.error_handlers` like below
@@ -96,6 +158,12 @@ PlanConsumer.enqueue_in(5.seconds, a: 1)
 ```
 
 ### Work with ActiveJob
+Config rails to use `:hutch` active_job adapter:
+```
+    config.active_job.queue_adapter     = :hutch
+    config.active_job.queue_name_prefix = 'ajd'
+```
+
 ```ruby
 class EmailJob < ApplicationJob
   queue_as :email
@@ -114,6 +182,13 @@ EmailJob.perform_later(user.id)
 EmailJob.set(wait: 5.seconds).perform_later(user.id)
 ```
 
+## Limits
+Because we use monkey patch to add some new features so if you direct use hutch cli and don't have some place to require
+hutch-schedule gem it won't be function.
+
+* When direct use hutch cli only have one hook for rails app to load external depencency so if the app is not an rails app right now will not working.
+* Hutch Config files must load during rails app initlization because monkey patch `Hutch::Config` need to load and then to work
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/wppurking/hutch-schedule. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
@@ -128,4 +203,3 @@ Use the repo: https://github.com/wppurking/hutch-schedule-demo
 
 # TODO
 * add cron job support
-* add unique job support

data/lib/hutch/enqueue.rb

@@ -16,6 +16,12 @@ module Hutch
         Hutch.publish(enqueue_routing_key, message)
       end
       
+      # enqueue unique message
+      def enqueue_uniq(uniq_key, message)
+        return false unless uniq_key_check(uniq_key)
+        enqueue(message)
+      end
+      
       # publish message at a delay times
       # interval: delay interval seconds
       # message: publish message
@@ -34,6 +40,11 @@ module Hutch
         Hutch::Schedule.publish(delay_routing_key, message, properties)
       end
       
+      def enqueue_uniq_in(uniq_key, interval, message, props = {})
+        return false unless uniq_key_check(uniq_key)
+        enqueue_in(interval, message, props)
+      end
+      
       # delay at exatly time point
       def enqueue_at(time, message, props = {})
         # compatible with with ActiveJob API
@@ -43,6 +54,17 @@ module Hutch
         enqueue_in(interval, message, props)
       end
       
+      def enqueue_uniq_at(uniq_key, time, message, props = {})
+        return false unless uniq_key_check(uniq_key)
+        enqueue_at(time, message, props)
+      end
+      
+      # check uniq_key is set or not
+      # expire time set for 24h
+      def uniq_key_check(uniq_key)
+        Hutch::Schedule.ns.set(uniq_key, "1", ex: 86400, nx: true)
+      end
+      
       # routing_key: the purpose is to send message to hutch exchange and then routing to the correct queue,
       # so can use any of them routing_key that the consumer is consuming.
       def enqueue_routing_key

data/lib/hutch/patch/config.rb

@@ -10,19 +10,24 @@ module Hutch
     # Hutch Schedule poller batch size
     number_setting :poller_batch_size, 100
     
-    # Ratelimit redis url
-    string_setting :ratelimit_redis_url, "redis://127.0.0.1:6379/0"
+    # Redis url for ratelimit and unique job
+    string_setting :redis_url, "redis://127.0.0.1:6379/0"
     
     # Ratelimit bucket interval
     number_setting :ratelimit_bucket_interval, 1
     
+    # Ratelimit redis backend reconnect attempts
+    number_setting :ratelimit_redis_reconnect_attempts, 10
+    
     initialize(
       worker_pool_size:  20,
       poller_interval:   1,
       poller_batch_size: 100,
       # @see Redis::Client
-      ratelimit_redis_url:       "redis://127.0.0.1:6379/0",
-      ratelimit_bucket_interval: 1
+      redis_url:                          "redis://127.0.0.1:6379/0",
+      ratelimit_bucket_interval:          1,
+      ratelimit_redis_reconnect_attempts: 10
     )
+    define_methods
   end
 end

data/lib/hutch/patch/worker.rb

@@ -19,15 +19,21 @@ module Hutch
         heartbeat_connection
         retry_buffer_queue
       end
-      @buffer_queue   = ::Queue.new
-      @batch_size     = Hutch::Config.get(:poller_batch_size)
-      @connected      = Hutch.connected?
+      
+      # The queue size maybe the same as channel[prefetch] and every Consumer have it's own buffer queue with the same prefetch size,
+      # when the buffer queue have the prefetch size message rabbitmq will stop push message to this consumer but it's ok.
+      # The consumer will threshold by the shared redis instace.
+      @buffer_queue = ::Queue.new
+      @batch_size   = Hutch::Config.get(:poller_batch_size)
+      @connected    = Hutch.connected?
     end
     
-    # 停止两个线程池
-    def shutdown
-      @message_worker.shutdown
+    # Stop a running worker by killing all subscriber threads.
+    # Stop two thread pool
+    def stop
       @timer_worker.shutdown
+      @message_worker.shutdown
+      @broker.stop
     end
     
     # Bind a consumer's routing keys to its queue, and set up a subscription to

data/lib/hutch/schedule.rb

@@ -30,7 +30,7 @@ module Hutch
         ActiveJob::QueueAdapters::HutchAdapter.register_actice_job_classes if defined?(ActiveJob::QueueAdapters::HutchAdapter)
         
         return if core.present?
-        Hutch.connect unless Hutch.connected?
+        Hutch.connect
         @core = Hutch::Schedule::Core.new(Hutch.broker)
         @core.connect!
       end
@@ -40,11 +40,28 @@ module Hutch
         @core = nil
       end
       
-      
       def core
         @core
       end
       
+      # redis with namespace
+      def ns
+        @redis ||= Redis::Namespace.new(:hutch, redis: Redis.new(
+          url: Hutch::Config.get(:redis_url),
+          # https://github.com/redis/redis-rb#reconnections
+          # retry 10 times total cost 10 * 30 = 300s
+          reconnect_attempts:  Hutch::Config.get(:ratelimit_redis_reconnect_attempts),
+          :reconnect_delay     => 3,
+          :reconnect_delay_max => 30.0,
+        ))
+      end
+      
+      # all Consumers that use threshold module shared the same redis instance
+      def redis
+        ns.redis
+      end
+      
+      
       def publish(*args)
         core.publish(*args)
       end

data/lib/hutch/schedule/version.rb

@@ -1,5 +1,5 @@
 module Hutch
   module Schedule
-    VERSION = "0.7.0"
+    VERSION = "0.7.2"
   end
 end

data/lib/hutch/threshold.rb

@@ -1,4 +1,7 @@
 require 'active_support/concern'
+require 'active_support/core_ext/object/blank'
+require 'active_support/core_ext/module/attribute_accessors'
+require 'hutch/schedule'
 require 'ratelimit'
 
 module Hutch
@@ -8,6 +11,10 @@ module Hutch
     
     # Add Consumer methods
     class_methods do
+      mattr_accessor :default_context, default: "default"
+      mattr_accessor :default_rate, default: 10
+      mattr_accessor :default_interval, default: 1
+      
       # 限流速度:
       #  context: 可选的上下文, 默认为 default
       #  rate: 数量
@@ -29,17 +36,17 @@ module Hutch
           raise "need args or block" if args.blank?
           raise "args need hash type" if args.class != Hash
           args.symbolize_keys!
-          @context  = args[:context].presence || "default"
-          @rate     = args[:rate]
-          @interval = args[:interval]
+          @context  = args[:context].presence || default_context
+          @rate     = args[:rate].presence || default_rate
+          @interval = args[:interval].presence || default_interval
         end
         # call redis.ping let fail fast if redis is not avalible
-        _redis.ping
+        Hutch::Schedule.redis.ping
         # redis: 传入设置的 redis
         # bucket_interval: 记录的间隔, 越小精度越大
         @rate_limiter = Ratelimit.new(self.name,
                                       bucket_interval: Hutch::Config.get(:ratelimit_bucket_interval),
-                                      redis:           _redis)
+                                      redis:           Hutch::Schedule.redis)
       end
       
       # is class level @rate_limiter _context exceeded?
@@ -47,29 +54,29 @@ module Hutch
       def ratelimit_exceeded?
         return false if @rate_limiter.blank?
         @rate_limiter.exceeded?(_context, threshold: _rate, interval: _interval)
+      rescue Redis::BaseError
+        # when redis cann't connect return exceeded limit
+        true
       end
       
       # 增加一次调用
       def ratelimit_add
         return if @rate_limiter.blank?
         @rate_limiter.add(_context)
+      rescue Redis::BaseError
+        nil
       end
       
       def _context
-        @block_given ? @threshold_block.call[:context] : @context
+        @block_given ? @threshold_block.call[:context].presence || default_context : @context
       end
       
       def _rate
-        @block_given ? @threshold_block.call[:rate] : @rate
+        @block_given ? @threshold_block.call[:rate].presence || default_rate : @rate
       end
       
       def _interval
-        @block_given ? @threshold_block.call[:interval] : @interval
-      end
-      
-      # all Consumers that use threshold module shared the same redis instance
-      def _redis
-        @@redis ||= Redis.new(url: Hutch::Config.get(:ratelimit_redis_url))
+        @block_given ? @threshold_block.call[:interval].presence || default_interval : @interval
       end
     end
   end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: hutch-schedule
 version: !ruby/object:Gem::Version
-  version: 0.7.0
+  version: 0.7.2
 platform: ruby
 authors:
 - Wyatt pan
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-04-12 00:00:00.000000000 Z
+date: 2020-04-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: hutch