tally_jobs 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 3b77ee7cc397f75d41288171fdda4c1b6afcfbd33f492b1a4b020ca32c8dc358
+  data.tar.gz: 44e3bf15cc99abe02cf3024bc1cf598fa6be7982fde5930b84f871eca64fa8f5
+SHA512:
+  metadata.gz: cb8455dcfde48404752169a07c0ebe7b100916ff6dd6455a11f5735f805836d3cd3ea91b6871f2fac51465257c1b5ed857df4212c8b6bd9ad0b676adb31cdfc9
+  data.tar.gz: d2442c12d194153c62a69f0f9408d5fc4832f4f101f9c7685955b3648fa5666376303a9e5e417cb38207ce1a4c4f834202e432bca4cf4303e81dcf300b69a4a6

data/README.md

@@ -0,0 +1,136 @@
+# TallyJobs
+
+Collect all params of the same jobs within an interval time then enqueue that job only one time.
+
+## Installation
+
+```ruby
+gem "tally_jobs"
+
+$ bundle install
+$ rails g tally_jobs:install
+```
+
+## Usage
+
+Assume that you have a job like this
+```ruby
+class ReportSpamCommentJob < ApplicationJob
+    queue_as :default
+
+    def perform(comment_id)
+        comment = Comment.find(comment_id)
+        Report.report_spam_comment(comment) if SpamDetective.check_spam(comment)
+    end
+end
+
+class CommentController < ApplicationController
+    # ...
+    def create
+        # ...
+        ReportSpamCommentJob.enqueue_to_tally(comment.id)
+        # ...
+    end
+
+    def update
+        # ...
+        ReportSpamCommentJob.perform_later(comment.id)
+        # ...
+    end
+    # ...
+end
+```
+
+In a peak time, a lot of comments are created and so a lot of jobs are enqueued. 
+If we gather all comment ids within a small interval time then enqueue only one job for them, so what we need is just one read query to fetch all comments, and one write query to report all spam comments.
+
+`tally_jobs` gem help you to do that:
+
+```ruby
+class CommentController < ApplicationController
+    # ...
+    def create
+        # ...
+        ReportSpamCommentJob.enqueue_to_tally(comment.id)
+        # ...
+    end
+    # ...
+end
+
+class ReportSpamCommentJob < ApplicationJob
+    queue_as :default
+
+    include TallyJobs::TallyData
+
+    def perform(*comment_ids)
+        comments = Comment.where(id: comment_ids) # one read query
+        if spams = SpamDetective.check_spams(comments)
+            Report.report_spam_comments(spams) # one write query
+        end
+    end
+end
+```
+
+The basic idea: 
+
+- you call `YouJob#enqueue_to_tally` to enqueue your job to a jobs-queue (in development, it is a `Thread::Queue`, in production it is `Redis List`).
+
+- `tally_jobs` will start a counter thread which, every interval time, will pop enqueued jobs, collect params list group by ActiveJob/ConfiguredJob class, then enqueue each job with its params collection.
+
+
+## Notes
+
+- `ActiveJob::ConfiguredJob` is counted separately from `ActiveJob`
+```ruby
+    ReportSpamCommentJob.enqueue_to_tally(1)
+    ReportSpamCommentJob.enqueue_to_tally(2)
+    ReportSpamCommentJob.set(wait_until: Date.tomorrow.noon).enqueue_to_tally(3)
+    ReportSpamCommentJob.set(wait_until: Date.tomorrow.noon).enqueue_to_tally(4)
+    # => enqueue 2 tally job
+    # one for ReportSpamCommentJob with [1,2]
+    # and one for [@job_class=ReportSpamCommentJob, @options={:wait_until=>Thu, 25 Apr 2024 12:00:00.000000000 UTC +00:00}] with [3,4]
+    #
+```
+
+- support perform in-batch
+
+```ruby
+class ReportSpamCommentJob < ApplicationJob
+    queue_as :default
+
+    include TallyJobs::TallyData
+
+    batch_size 100
+
+    def perform(one_hundred_comment_ids)
+    end
+end
+```
+
+- call `TallyJobs#stop` to stop counter thread, call `TallyJobs#restart` to restart counter thread
+- in tests, just stop counter thread before all test cases, if you want to test job enqueued, you could start/flush/stop counter thread on each test case:
+    
+    ```ruby
+    expect {
+        TallyJobs.restart
+            
+        ReportSpamCommentJob.enqueue_to_tally(3)
+        ReportSpamCommentJob.enqueue_to_tally(4)
+        ReportSpamCommentJob.enqueue_to_tally(5)
+        TallyJobs.flush # force collect and enqueue jobs
+        # expect ...
+        
+        TallyJobs.stop # this is also call flush
+    }.to have_enqueued_job(ReportSpamCommentJob).with([3,4,5])
+    ```
+
+## Todo
+
+- handling back pressure
+- support multiple tally jobs queues base on ActiveJob `queue_as`, set `interval-time` for each queue base on it's priority (higher priority, smaller interval time)
+- support ActionMailer ???
+
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/tally_jobs.

data/Rakefile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec

data/lib/generators/tally_jobs/install_generator.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+require 'rails/generators/base'
+
+module TallyJobs
+    module Generators
+        class InstallGenerator < Rails::Generators::Base
+            source_root File.expand_path("../../templates", __FILE__)
+
+            def copy_initializer
+                copy_file "tally_jobs.rb", "config/initializers/tally_jobs.rb"
+            end
+        end
+    end
+end

data/lib/generators/templates/tally_jobs.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+TallyJobs.setup do |config|
+    config.interval = 120 # seconds
+    config.logger = Rails.logger
+    # config.counter_store = :memory_counter_store
+    config.counter_store = { store: :redis_counter_store, redis: Redis.new(host: "localhost") } if Rails.env.production?
+end

data/lib/tally_jobs/configs.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+require_relative "./counter_store/memory_counter_store"
+require_relative "./counter_store/redis_counter_store"
+
+module TallyJobs
+    class Configs
+        attr_accessor :interval, :logger, :redis
+
+        def redis=(_redis)
+            redis = _redis
+        end
+
+        def counter_store=(store)
+            JobsCounter.store = \
+                case store
+                when :memory_counter_store
+                    TallyJobs::CounterStore::MemoryCounterStore.new
+                when Hash
+                    case store[:store]
+                    when :redis_counter_store
+                        TallyJobs::CounterStore::RedisCounterStore.new(**store.except(:store))
+                    else
+                        store.delete(:store).to_s.classify.constantize.new(**store)
+                    end
+                else
+                    store.to_s.classify.constantize.new
+                end
+        end
+    end
+end

data/lib/tally_jobs/configured_job.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+module TallyJobs
+    module ConfiguredJob
+        def enqueue_to_tally(*params)
+            TallyJobs::JobsCounter.store.enqueue(
+                [self.instance_variable_get(:@job_class), self.instance_variable_get(:@options)], 
+                *params
+            )
+        end
+    end
+end
+
+# to support enqueue_to_tally preconfigured jobs
+require 'active_job'
+::ActiveJob::ConfiguredJob.include TallyJobs::ConfiguredJob

data/lib/tally_jobs/counter_store/memory_counter_store.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+require_relative "../counter_store.rb"
+
+module TallyJobs::CounterStore
+    class MemoryCounterStore < Base
+        # in-memory job queue
+        JOBS_QUEUE = Thread::Queue.new
+        
+        def enqueue(job_clazz, *params)
+            JOBS_QUEUE.enq([job_clazz, *params])
+        end
+
+        def dequeue(n = 1)
+            JOBS_QUEUE.deq
+        end
+
+        delegate :empty?, :clear, to: "TallyJobs::CounterStore::MemoryCounterStore::JOBS_QUEUE"
+    end
+end

data/lib/tally_jobs/counter_store/redis_counter_store.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+require_relative "../counter_store.rb"
+
+module TallyJobs::CounterStore
+    class RedisCounterStore < Base
+        attr_reader :redis
+
+        def initialize(redis:)
+            @redis = redis
+        end
+
+        def enqueue(job_clazz, *params)
+            @redis.rpush(KEY, Marshal.dump([job_clazz, *params]))
+        end
+
+        def dequeue(n = 1)
+            Marshal.load(@redis.lpop(KEY))
+        end
+
+        def empty?
+            @redis.llen(KEY).zero?
+        end
+
+        def clear
+            @redis.del(KEY)
+        end
+
+        private
+            
+            KEY = "tally-jobs-queue".freeze
+    end
+end

data/lib/tally_jobs/counter_store.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+module TallyJobs::CounterStore
+    class Base
+        def enqueue(job_clazz, *params)
+        end
+
+        def dequeue(n = 1)
+        end
+
+        def empty?
+        end
+
+        def clear
+        end
+    end
+end

data/lib/tally_jobs/engine.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+module TallyJobs
+    class Engine < ::Rails::Engine
+    end
+end

data/lib/tally_jobs/jobs_counter.rb

@@ -0,0 +1,39 @@
+# frozen_string_literal: true
+
+require_relative "./counter_store/memory_counter_store"
+
+module TallyJobs
+    class JobsCounter
+        cattr_accessor :store, default: TallyJobs::CounterStore::MemoryCounterStore.new
+
+        def self.collect_then_perform_later
+            groups = Hash.new { |h, k| h[k] = [] }
+            until store.empty?
+                job_clazz, *params = store.dequeue
+                groups[job_clazz] << (params.size == 1 ? params[0] : params)
+            end
+
+            groups.each do |job_clazz, params_list|
+                batch_size = nil
+
+                if job_clazz.is_a? Array
+                    job_clazz, options = *job_clazz
+                    batch_size = job_clazz._batch_size
+                    job_clazz = job_clazz.set(**options)
+                else
+                    batch_size = job_clazz._batch_size
+                end
+
+                next if job_clazz.nil?
+
+                if batch_size.nil?
+                    job_clazz.perform_later(params_list)
+                else
+                    params_list.each_slice(batch_size).each do |slice_params|
+                        job_clazz.perform_later(slice_params)
+                    end
+                end
+            end
+        end
+    end
+end

data/lib/tally_jobs/tally_data.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+module TallyJobs
+    module TallyData
+        def self.included(base)
+            base.extend ClassMethods
+            base.class_eval do
+                cattr_accessor :_batch_size
+
+                def self.batch_size(size)
+                    self._batch_size = size
+                end
+            end
+        end
+
+        module ClassMethods
+            def enqueue_to_tally(*params)
+                TallyJobs::JobsCounter.store.enqueue(self, *params)
+            end
+        end
+    end
+end

data/lib/tally_jobs/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module TallyJobs
+  VERSION = "0.0.1"
+end

data/lib/tally_jobs.rb

@@ -0,0 +1,70 @@
+# frozen_string_literal: true
+
+require_relative "tally_jobs/version"
+require_relative "tally_jobs/engine"
+require_relative "tally_jobs/configs"
+require_relative "tally_jobs/jobs_counter"
+require_relative "tally_jobs/tally_data"
+require_relative "tally_jobs/configured_job"
+
+module TallyJobs
+  class Error < StandardError; end
+
+  mattr_accessor :runnable, default: true
+  mattr_reader :configs, default: TallyJobs::Configs.new
+  class << self
+    def setup
+      yield configs
+      
+      TallyJobs.start
+    end
+  end
+
+  module_function
+  
+  # start a thread to flush in-memory enqueued jobs
+  # each {interval} time
+  MUTEX = Mutex.new
+  def start
+    MUTEX.synchronize do
+      @tally_thread = nil if !@tally_thread&.alive?
+
+      @tally_thread ||= Thread.new do
+        TallyJobs.configs.logger&.info("[TallyJobs] started ...")
+        while true
+          break unless TallyJobs.runnable
+          sleep(TallyJobs.configs.interval || 300)
+
+          JobsCounter.collect_then_perform_later
+        end
+      end
+    end
+  end
+
+  def flush
+    TallyJobs.configs.logger&.info("[TallyJobs] flushing ...")
+    JobsCounter.collect_then_perform_later
+  end
+
+  def stop
+    flush
+
+    TallyJobs.runnable = false
+    if @tally_thread && @tally_thread.alive?
+      @tally_thread.wakeup
+      @tally_thread.join
+    end
+    TallyJobs.configs.logger&.info("[TallyJobs] stop ...")
+  end
+
+  def restart
+    stop
+    TallyJobs.runnable = true
+    start
+  end
+
+  # force flush all in-memory enqueued jobs before exit
+  at_exit do
+    TallyJobs.stop
+  end
+end

metadata

@@ -0,0 +1,145 @@
+--- !ruby/object:Gem::Specification
+name: tally_jobs
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- theforestvn88
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2024-04-24 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: activejob
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: railties
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec-rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: redis
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Collect all params of the same jobs within an interval time then enqueue
+  that job only one time.
+email:
+- theforestvn88@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- Rakefile
+- lib/generators/tally_jobs/install_generator.rb
+- lib/generators/templates/tally_jobs.rb
+- lib/tally_jobs.rb
+- lib/tally_jobs/configs.rb
+- lib/tally_jobs/configured_job.rb
+- lib/tally_jobs/counter_store.rb
+- lib/tally_jobs/counter_store/memory_counter_store.rb
+- lib/tally_jobs/counter_store/redis_counter_store.rb
+- lib/tally_jobs/engine.rb
+- lib/tally_jobs/jobs_counter.rb
+- lib/tally_jobs/tally_data.rb
+- lib/tally_jobs/version.rb
+homepage: https://github.com/theforestvn88/tally_jobs.git
+licenses: []
+metadata:
+  homepage_uri: https://github.com/theforestvn88/tally_jobs.git
+  source_code_uri: https://github.com/theforestvn88/tally_jobs.git
+  changelog_uri: https://github.com/theforestvn88/tally_jobs.git
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 1.0.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.4.10
+signing_key: 
+specification_version: 4
+summary: Collect all params of the same jobs within an interval time then enqueue
+  that job only one time.
+test_files: []