belated 0.7.0 → 0.8.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 625adf33d85f1e94d4d7416db02a50e11fe66bfeb5fa265fa4de2c9b6f1c0750
-  data.tar.gz: 663fa5c0fcb7903b66324723a8cd19a771d596a88a396974d80060bf8c0bfd4a
+  metadata.gz: c440417194e561ef7dacc37228c9025afe230dca489d7a80f5478e8d38f75231
+  data.tar.gz: 8e2d9c85906dc9e25a201388a7a8fd155406704cd573d7a7599d3e3af9344f55
 SHA512:
-  metadata.gz: e14413f88843ed9c80391987c5891f821ce0eb43e2759d5be034bb17a9cda78f4de7a67b97f44af933013dd4aef39e0d23d91df9ecfbbf3dbead38427a925450
-  data.tar.gz: aef2baada5a40a7da090e71e1c490d6f901e5178bd5be05001fb53c2863e9b03b86f9b997a53ced594d40aad8eb8700e25419b1378eba13b444b3b62f48d1b65
+  metadata.gz: ddfb739688a3efefa4c4c503c64008badd5b5befba1a96ee2d9817af7e84615429aaa888266712551e06451aae5c5b9c5c1bd5e5ac3ac0d392fb0b02470c9d34
+  data.tar.gz: 452b13016e6c1d08d6784d96e6234750eb9741b774ee1e8a9a08721d0e0d53bf49d9f9442647cec324e52e818fb230591cae136f67b88b98fabb8cf07a3ecfe5

data/.github/workflows/main.yml

@@ -7,9 +7,9 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: [ubuntu-latest, macos-latest]
+        os: [ubuntu-latest]
         # Due to https://github.com/actions/runner/issues/849, we have to use quotes for '3.0'
-        ruby: [2.7, '3.0']
+        ruby: [2.6, 2.7, '3.0']
     runs-on: ${{ matrix.os }}
     steps:
     - uses: actions/checkout@v2

data/CHANGELOG.md

@@ -1,5 +1,19 @@
 ## [Unreleased]
 
+## [0.8.3] - 2021-09-10
+- Add environment to the names of the pstore files. This way you won't get your development, staging, and production jobs mixed up. (I was getting my test and dev jobs mixed up)
+
+## [0.8.2] - 2021-09-09
+- Fixed a bug where the adapter was not defined when loading jobs from the PStore file. 
+
+## [0.8.1] - 2021-09-09
+- Now you can delete jobs from the future jobs queue. This is useful if you want to delete a job that is scheduled for a future date.
+
+## [0.8] - 2021-09-08
+- Using PStore for future jobs backup and job history. Job history is rotated daily, future jobs cannot at the moment. 
+- PStore has ultrasafe mode, should test whether that makes things very slow. If not, might be worth using it for the peace of mind. Or at least have it as an option. 
+- ActiveJob support is a bit more natural now code-wise. 
+
 ## [0.7] - 2021-09-04
 - ActiveJob support! Retries, exception rescuing should work as expected. 
 - Second Moderna jab took me out for a while... sorry for the long wait. 

data/Gemfile.lock

@@ -1,9 +1,10 @@
 PATH
   remote: .
   specs:
-    belated (0.7.0)
+    belated (0.8.3)
       drb
       dry-configurable
+      pstore
       ruby2_keywords
       sorted_set
 
@@ -111,6 +112,7 @@ GEM
     parallel (1.20.1)
     parser (3.0.2.0)
       ast (~> 2.4.1)
+    pstore (0.1.1)
     racc (1.5.2)
     rack (2.2.3)
     rack-test (1.1.0)

data/README.md

@@ -2,11 +2,9 @@
 
 [![CodeFactor](https://www.codefactor.io/repository/github/sampokuokkanen/belated/badge)](https://www.codefactor.io/repository/github/sampokuokkanen/belated) [![Gem Version](https://badge.fury.io/rb/belated.svg)](https://badge.fury.io/rb/belated)
 
-This is Belated, a new Ruby backend job library! It supports running procs, lambdas and classes in the background. To deal with restarts, it uses YAML to load the queue into a file, which it then calls at startup to find the previous jobs. There is no way in Ruby to save procs or lambdas to a file, so they are discarded when the process restarts.
+This is Belated, a new Ruby backend job library! It supports running procs, lambdas and classes in the background. To deal with restarts, it uses YAML for the current jobs in the queue waiting to be processed and PStore for the future jobs to load the queues into a file, which it then calls at startup to find the previous jobs. There is no way in Ruby to save procs or lambdas to a file, so they are discarded when the process restarts.
 
-Belated uses the Ruby Queue class, so it's First In, First Out (FIFO). 
-
-Note that Belated used to be called HardWorker. That name was already in use in Sidekiq documentation and a bit too generic anyway. 
+Belated uses the Ruby Queue class, so it's First In, First Out (FIFO), unless of course you want to run the job in the future. In that case the order is decided by the time the job is scheduled to be executed. 
 
 It uses dRuby to do the communication! Which is absolute great. No need for Redis or PostgreSQL, just Ruby standard libraries.
 
@@ -16,14 +14,14 @@ Can be used if you're on a normal instance such as EC2 or Digital Ocean drop. No
 
 TODO LIST:
 
-- Use GDBM for queue storage? That way could maybe get rid of YAML dumping and make things a bit safer. Not ordered though, so maybe keep a list of the jobs as YAML and update it sometimes? Just as backup. Or RocksDB? Would need to be configurable if you don't have something installed.
-- Have a web UI.
-- Have a job history
-- Do some performance testing.
+- Have a web UI with job history.
 - Deploy a Rails app to production that is using Belated
   and mention it in the readme. (Capistrano support?)
-  ([Wasurechatta](https://wasurechatta.com/))
-- Add a section telling people to use Sidekiq if they can
+  ([Wasurechatta](https://wasurechatta.com/) deployed, still need to setup Capistrano)
+
+# Why not Sidekiq? 
+
+If you can, definitely use Sidekiq!!! Belated is supposed to be used if you can't get anything else to work. Like if you want to use SQLite in a Rails app and don't want to have Redis running. Or maybe you just want to run procs in the background? 
 
 ## Installation
 
@@ -76,8 +74,21 @@ client.perform_belated(DumDum.new, at: Time.now + 5 * 60)
 client.perform_belated(DumDum.new, max_retries: 3) # default 5
 ```
 
-Belated runs on localhost, port 8788. 
-Going to make that an option in the future.
+You can also fetch jobs from the future jobs queue:
+
+```ruby
+job = client.perform_belated(proc { 0 / 0 }, at: Time.now + 5 * 60)
+job = Belated.find job.id # Find the job if it's in the future queue
+# Oh no, that job looks a bit weird!
+# Let's delete it:
+client.perform_belated(
+  Belated.delete job.id
+)
+# Yeah... currently you have to send the command through the client like this as a job. :/
+# Maybe the client should handle the deletion?
+```
+
+Belated runs on localhost, port 8788 by default, but the port is configurable, see below. 
 
 ## Rails
 

data/belated.gemspec

@@ -34,6 +34,7 @@ Gem::Specification.new do |spec|
   spec.add_dependency 'drb'
   spec.add_dependency 'dry-configurable'
   spec.add_dependency 'ruby2_keywords'
+  spec.add_dependency 'pstore'
   spec.add_dependency 'sorted_set'
   spec.add_development_dependency 'byebug'
 

data/lib/active_job/queue_adapters/belated_adapter.rb

@@ -22,6 +22,13 @@ module ActiveJob # :nodoc:
         Rails.logger.info "Belated got job #{job} to be performed at #{Time.at(timestamp)}"
         instance.perform_belated(job, at: timestamp, active_job: true)
       end
+
+      # JobWrapper that overwrites perform for ActiveJob
+      class JobWrapper < Belated::JobWrapper
+        def perform
+          Base.execute job.serialize
+        end
+      end
     end
   end
 end

data/lib/belated/client.rb

@@ -101,7 +101,12 @@ class Belated
     def wrap_job(job, at:, max_retries:, active_job:)
       return job if job.is_a?(JobWrapper)
 
-      JobWrapper.new(job: job, at: at, max_retries: max_retries, active_job: active_job)
+      wrapper = if active_job
+                  ActiveJob::QueueAdapters::BelatedAdapter::JobWrapper
+                else
+                  JobWrapper
+                end
+      wrapper.new(job: job, at: at, max_retries: max_retries, active_job: active_job)
     end
 
     def drb_connected?

data/lib/belated/job_wrapper.rb

@@ -13,21 +13,23 @@ class Belated
   class JobWrapper
     include Comparable
     include Logging
-    attr_accessor :retries, :max_retries, :id, :job, :at, :completed, :proc_klass, :error, :active_job
+    attr_accessor :retries, :max_retries, :id, :job, :at,
+                  :completed, :proc_klass, :error, :active_job
 
     def initialize(job:, max_retries: 5, at: nil, active_job: false)
+      raise 'JobError' unless job.respond_to?(:call) || job.respond_to?(:perform)
+
       self.retries = 0
       self.max_retries = max_retries
       self.id = job.respond_to?(:job_id) ? job.job_id : SecureRandom.uuid
       self.job = job
       self.at = at
-      self.completed = false
       self.proc_klass = job.instance_of?(Proc)
       self.active_job = active_job
     end
 
     def <=>(other)
-      at <=> other.at
+      at <=> (other&.at || other&.scheduled_at)
     end
 
     # rubocop:disable Lint/RescueException
@@ -37,7 +39,7 @@ class Belated
       resp
     rescue Exception => e
       case e.class
-      when Interrupt, SignalException
+      when Interrupt, SignalException, NoMethodError
         raise e
       else
         retry_job(e)
@@ -47,9 +49,7 @@ class Belated
 
     # rubocop:enable Lint/RescueException
     def execute
-      if active_job
-        ActiveJob::Base.execute job.serialize
-      elsif job.respond_to?(:call)
+      if job.respond_to?(:call)
         job.call
       elsif job.respond_to?(:arguments)
         job.perform(*job.arguments)

data/lib/belated/queue.rb

@@ -4,7 +4,7 @@ require 'belated/job'
 require 'belated/logging'
 require 'belated/job_wrapper'
 require 'sorted_set'
-
+require 'pstore'
 class Belated
   # Job queues that Belated uses.
   # queue is the jobs that are currenly
@@ -13,7 +13,7 @@ class Belated
   # to be added to queue at some point in the future.
   class Queue
     include Logging
-    attr_accessor :future_jobs
+    attr_accessor :future_jobs, :future_jobs_db
 
     FILE_NAME = 'belated_dump'
 
@@ -21,6 +21,24 @@ class Belated
       @queue = queue
       @mutex = Mutex.new
       self.future_jobs = future_jobs
+      self.future_jobs_db = PStore.new("future_jobs_#{Belated.environment}.pstore", true) # pass true for thread safety
+    end
+
+    def enqueue_future_jobs
+      loop do
+        job = future_jobs.min
+        if job.nil?
+          sleep Belated.heartbeat
+          next
+        end
+        if job.at <= Time.now.to_f
+          delete_job(job)
+          push(job)
+        end
+      rescue DRb::DRbConnError
+        error 'DRb connection error!!!!!!'
+        log stats
+      end
     end
 
     def push(job)
@@ -30,6 +48,7 @@ class Belated
       else
         @mutex.synchronize do
           @future_jobs << job
+          insert_into_future_jobs_db(job) unless job.proc_klass
         end
       end
     end
@@ -52,16 +71,16 @@ class Belated
     end
 
     def load_jobs
-      log "reloading... if file exists #{File.exist?(FILE_NAME)}"
+      future_jobs_db.transaction(true) do
+        future_jobs_db.roots.each do |id|
+          future_jobs << future_jobs_db[id]
+        end
+      end
       return unless File.exist?(FILE_NAME)
 
       jobs = YAML.load(File.binread(FILE_NAME))
       jobs.each do |job|
-        if job.at && job.at > Time.now.to_f
-          future_jobs.push(job)
-        else
-          @queue.push(job)
-        end
+        @queue.push(job)
       end
       File.delete(FILE_NAME)
     end
@@ -73,12 +92,6 @@ class Belated
           class_array << klass
         end
       end
-      future_jobs.each do |_job|
-        unless proc_or_shutdown?(klass = future_jobs.pop)
-          class_array << klass
-        end
-      end
-
       pp File.open(FILE_NAME, 'wb') { |f| f.write(YAML.dump(class_array)) }
     end
 
@@ -86,10 +99,32 @@ class Belated
       true
     end
 
+    def find(job_id)
+      job = nil
+      future_jobs_db.transaction(true) do
+        job = future_jobs_db[job_id]
+      end
+      job = future_jobs.find { |j| j.id == job_id } if job.nil?
+      job
+    end
+
+    def delete_job(job)
+      log "Deleting #{future_jobs.delete(job)} from future jobs"
+      future_jobs_db.transaction do
+        future_jobs_db.delete(job.id)
+      end
+    end
+
     private
 
     def proc_or_shutdown?(job)
       job.is_a?(Symbol) || job.job.instance_of?(Proc)
     end
+
+    def insert_into_future_jobs_db(job)
+      future_jobs_db.transaction do
+        future_jobs_db[job.id] = job
+      end
+    end
   end
 end

data/lib/belated/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 class Belated
-  VERSION = '0.7.0'
+  VERSION = '0.8.3'
 end

data/lib/belated/worker.rb

@@ -1,4 +1,6 @@
 require_relative 'logging'
+require 'pstore'
+
 class Belated
   # The worker class that actually gets the jobs from the queue
   # and calls them. Expects the jobs to be procs or
@@ -20,9 +22,27 @@ class Belated
 
         log "Worker #{@number} got job: #{job.inspect}"
         log job.perform
+        history_insert(job) unless job.proc_klass || !job.completed
       rescue DRb::DRbConnError, Errno::ECONNREFUSED, RangeError => e
         log e
       end
     end
+
+    private
+
+    def history_insert(job)
+      store.transaction do
+        store[job.id] = job
+      end
+    rescue StandardError => e
+      error e
+    end
+
+    def store
+      today = Time.now.strftime('%F')
+      return @store if @store&.path&.include?(today)
+
+      @store = PStore.new("history_#{Belated.environment}-#{today}.pstore", true)
+    end
   end
 end

data/lib/belated.rb

@@ -22,7 +22,6 @@ class Belated
   extend Dry::Configurable
   include Logging
   include Singleton unless $TESTING
-  @@queue = Belated::Queue.new
 
   setting :rails, true
   setting :rails_path, '.'
@@ -36,6 +35,7 @@ class Belated
   setting :heartbeat, 1, reader: true
   setting :client_heartbeat, 5, reader: true
   URI = "druby://#{Belated.host}:#{Belated.port}"
+  @@queue = Belated::Queue.new
 
   # Since it's running as a singleton, we need something to start it up.
   # Aliased for testing purposes.
@@ -51,7 +51,7 @@ class Belated
     connect!
     banner_and_info
     trap_signals
-    enqueue_future_jobs
+    @@queue.enqueue_future_jobs
   end
   alias initialize start
 
@@ -89,29 +89,13 @@ class Belated
     require File.expand_path("#{Belated.config.rails_path}/config/environment.rb")
     require 'rails/all'
     require 'belated/rails'
+    require 'active_job/queue_adapters/belated_adapter'
   end
 
   def rails?
     Belated.config.rails
   end
 
-  def enqueue_future_jobs
-    loop do
-      job = @@queue.future_jobs.min
-      if job.nil?
-        sleep Belated.heartbeat
-        next
-      end
-      if job.at <= Time.now.to_f
-        log "Deleting #{@@queue.future_jobs.delete(job)} from future jobs"
-        @@queue.push(job)
-      end
-    rescue DRb::DRbConnError
-      error 'DRb connection error!!!!!!'
-      log stats
-    end
-  end
-
   def reload
     log 'reloading...'
     @@queue.load_jobs
@@ -159,7 +143,12 @@ class Belated
 
   class << self
     def find(job_id)
-      @@queue.future_jobs.find { |job| job.id == job_id }
+      @@queue.find(job_id)
+    end
+
+    def delete(job_id)
+      job = find(job_id)
+      @@queue.delete_job(job)
     end
 
     def kill_and_clear_queue!

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: belated
 version: !ruby/object:Gem::Version
-  version: 0.7.0
+  version: 0.8.3
 platform: ruby
 authors:
 - Sampo Kuokkanen
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2021-09-04 00:00:00.000000000 Z
+date: 2021-09-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: drb
@@ -52,6 +52,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: pstore
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: sorted_set
   requirement: !ruby/object:Gem::Requirement