orbitalqueue 0.0.1 → 0.0.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1209a43eaeb6c2de53893b2c58ee64c42f7a14b8339f750443124d0a0c26fa08
-  data.tar.gz: 9cd589726e3198cb7ab20676b7c1151eea9331106653a5ffa250c40a2adae83e
+  metadata.gz: 0ebd1ce38a6bd9bdc2acd6f36571966aeddddef703cedb97136d67b8cad64316
+  data.tar.gz: '061397dd5691647ddb49753f48d7f931fdc17840b4bda68857b3f2f1b1d88431'
 SHA512:
-  metadata.gz: 9fcb2e82342484b0334fc4e95469e017ba633211547e93c4f171fbe6861d0403b22c7a07f66f14a30c3e756f455c65737671f91e6090822f51e1b91d6b38992e
-  data.tar.gz: 9235c90069bdabbbd359499914a92599ce3782329369980e9ade49b630619f4f04abd6ebf32e19baece2c4a9b28b124f33e5c00320455361e070671f13bb6753
+  metadata.gz: 37c7cd635345003b83446d3b82ce7036b91a0b3ba30913d41d62f7b06a0c618e7f6648784aab2c17661eb837b96330abedb06751a8db54f781587ae529daf2de
+  data.tar.gz: 3ee034e1f5d6ef47fb49ce744ea2355bf69ce54d8571c94436c1c092a04c448423e8ea2f01435eda36eecebf04e3a07980a45f66a94157895730d684e99faf71

data/README.md

@@ -1,4 +1,4 @@
-# orbital-queue
+# Orbital Queue
 
 [![CI](https://github.com/reasonset/orbital-queue/actions/workflows/ci.yml/badge.svg)](https://github.com/reasonset/orbital-queue/actions/workflows/ci.yml)
 
@@ -39,18 +39,282 @@ Since enqueued objects are serialized using `Marshal` and persisted as files, ca
 require 'orbitalqueue'
 
 queue = OrbitalQueue.new("/path/to/queue")
-data = queue.pop
+item = queue.pop
+data = item.data
 
 # Something, something...
 
-queue.complete data[:queue_id]
+item.complete
 ```
 
 Calling `#pop` retrieves a single item from the queue in no particular order.
 
+```ruby
+queue = OrbitalQueue.new("/path/to/queue")
+item = queue.pop!
+
+# Something, something...
+```
+
 The retrieved item enters a checkout state, and must be finalized by calling `#complete` once processing is finished.
 
 If guaranteed completion is not required and the item should be removed immediately upon retrieval, use `#pop!` instead.
 
-Both `#pop` and `#pop!` return a queue item as a `Hash`. If the original object was not a `Hash`, it will be stored under the `:data` key. Regardless of the original type, the queue ID is always stored under the `:queue_id` key.
+Both `#pop` and `#pop!` methods returns `OrbitalQueue::QueueObject` object.
+You can access queue data via `OrbitalQueue::QueueObject#data`.
+
+When calling `#complete`, `#destruct`, or `#defer` directly on `OrbitalQueue`, `queue_id` must be given.  
+If you use these methods via `OrbitalQueue::QueueObject`, the `queue_id` is internally handled and can be omitted.
+
+### Dequeue with block
+
+You can call `#pop` with a block.
+
+When call with block, block is called with queue data as an argument.
+
+queue item automatically complete when blocke ends without error.
+
+The `#pop` method can be called with an optional block.  
+When used this way, the block is invoked with the item's data.  
+After successful execution (without exceptions), the item is considered complete and removed from the queue.
+
+```ruby
+queue = OrbitalQueue.new("/path/to/queue")
+queue.pop do |data|
+  #...
+end
+```
+
+### Dequeue loop
+
+While technically possible to loop over `#pop`, it doesn't support block-based iteration.
+
+Use `#each` when you want to process items with a block in a loop.  
+It provides a clean and idiomatic Ruby style for sequential processing.
+
+For full control over each item, use `#each_item`, which yields a `QueueObject` instead of just the data.
+
+`#each_item` iterates over queue items as `QueueObject` instances, not raw data.  
+This allows direct control over each job—for example, deferring its execution using `#defer`.
+
+Unlike `#each`, which automatically marks items as complete after the block runs,  
+`#each_item` exposes queue control for cases where completion isn't guaranteed or deferred handling is needed.
+
+### Job deferral
+
+`OrbitalQueue` supports job deferral, enabling queue items to be scheduled for retry or postponed execution with precise control.
+
+Calling `#defer` transitions a queue item into the deferred state.  
+This moves the item's file into the `.defer` directory and creates a retry metadata file in `.retry`.
+
+Once an item has been deferred, it is associated with retry information, referred to as `retry_data`.  
+This is a `Hash` object that tracks rescheduling behavior.
+
+`#defer` uses `retry_data` to determine whether the item can be retried, including retry count limits.  
+When called with a block, `#defer` yields `retry_data` as an argument, allowing custom modifications.
+
+Regardless of how it's called, `retry_data` is persisted after the method returns.  
+To control deferral behavior, modify values inside `retry_data`—typically by changing the `:until` field (Unix timestamp).  
+This field specifies when the item should become eligible for re-queueing, making it ideal for implementing backoff strategies.
+
+```ruby
+queue.each_item do |item|
+  begin
+    # Something...
+  rescue
+    item.defer(Time.now + 300)     # Retry after 5 minutes.
+  end
+end
+```
+
+with block:
+
+```ruby
+queue.each_item do |item|
+  begin
+    # Something...
+  rescue
+    item.defer do |retry_item|
+      if retry_item[:count] > 5
+        item.destruct
+      else
+        retry_item[:until] = Time.now + 300 * retry_item[:count]
+      end
+    end
+  end
+end
+```
+
+The `#defer` method's main role is to move the queue item file into the `.defer` directory.  
+Since `OrbitalQueue` operates without a server, it cannot scan `.defer` efficiently or restore deferred items automatically.
+
+Aside from the internal keys `:count` and `:until`, all other values in `retry_data` are preserved as-is.  
+You can freely store custom metadata inside it—such as failure reasons or backoff parameters.
+
+The `#destruct` method removes all files associated with a queue item and raises `OrbitalQueue::ItemDestructed`.  
+This exception is caught inside `#defer`, allowing `#destruct` to abort the entire deferral process.
+
+⚠️ Do not rescue `OrbitalQueue::ItemDestructed` within a `#defer` block.  
+If the block completes normally after destruction, queue integrity may be violated.
+
+The `#archive` method creates a Marshal-serialized file under `.archive`, containing the original data, its `retry_data`, and an `archiveinfo` hash.  
+After archiving, it calls `#destruct` to remove the live queue item.
+
+Archived files are never accessed by OrbitalQueue itself.
+
+Note: Because `archive` discards in-memory `retry_data`, you cannot modify it before archiving.  
+Instead, extra metadata should be passed as arguments to `archive` and will be merged into `archiveinfo`.
+
+```ruby
+queue.each_item do |item|
+  begin
+    #...
+  rescue
+    item.defer do |retry_data|
+      if retry_data[:count] > 5
+        item.archive({reason: "Host timeout"})
+      end
+    end
+  end
+end
+```
+
+### Resume deferred job
+
+Deferred queue items must be manually restored using the `resume` method.  
+This method is typically executed by a separate worker from the one handling regular queue operations.
+
+It is defined as an instance method:
+
+```ruby
+queue = OrbitalQueue.new("/path/to/queue")
+queue.resume
+```
+
+For convenience, resume can also be called as a class method:
+
+```ruby
+OrbitalQueue.resume("/path/to/queue")
+```
+
+# About Orbital Design
+
+## Description
+
+Orbital Design is a programming pattern optimized for distributed systems.  
+It is especially well-suited to environments where:
+
+* New data constantly arrives without pause
+* Processing workloads vary in complexity and demand asymmetric distribution
+* Systems start small but must scale seamlessly to clustered deployments
+
+## Philosophy
+
+Orbital Design distinguishes between "agents" and "workers".  
+In most cases, an agent refers to a program, while a worker is a process.
+
+The core principle is that **workers only need to care about what they do**.  
+Upon starting, a worker picks a single available job prepared for it and executes it—no coordination or negotiation required.
+
+This behavior mirrors that of individuals in a larger society, or cells within a living organism.  
+Each unit performs its specific role independently.
+
+This philosophy is deeply aligned with the Unix principle:  
+_"Do one thing, and do it well."_
+
+## Core Rules of Orbital Design
+
+Orbital Design defines a set of principles to preserve decoupling, clarity, and safety in distributed systems:
+
+- *Agents must remain small and focused*.  
+  Each agent is responsible for doing one thing, and doing it well.
+
+- *Workers must not access data unrelated to their task*, nor inspect other workers' state or progress.
+
+- *Write access to a database or dataset must be held by exactly one agent*.  
+  This prevents conflicting updates and maintains integrity.
+
+- *Deletion from a database may only be performed by:*
+  - A worker with exclusive read access to the data, or
+  - A sweeper worker that receives notifications from all readers
+
+- *Agents must not block on I/O*.  
+  Blocking input/output disrupts concurrency and undermines distributed fairness.
+
+## Benefits of Orbital Design
+
+### Ease of Implementation
+
+Each program is small and focused, with clearly defined responsibilities.  
+Because agents cannot access global state and avoid blocking operations, race conditions are structurally prevented.
+
+This allows each unit to concentrate solely on its task—no need to worry about concurrency or system state.
+
+## Simplicity
+
+Orbital Design requires minimal complexity.  
+It does not depend on heavy frameworks or advanced techniques.
+
+It can be fully implemented using standard OS features such as file systems, processes, and signals.  
+No special measures are needed to achieve scalability.
+
+## Language Agnosticism
+
+Programs are isolated and do not interfere with one another.  
+This allows you to implement each agent in any language that suits the task.
+
+You can choose a language based on convenience, libraries, or performance.
+Critical paths can be written in C, C++, Rust, or Nim as needed, while simpler agents may use scripting languages.
+
+Even agents with similar functionality can be written in different languages depending on input format or operational context.
+
+## Parallelism and Decomposition
+
+Restricted I/O paths eliminate contention during parallel execution.  
+By following the design pattern, concurrency becomes straightforward.
+
+No locking or synchronization is required—so parallel processing not only becomes easier to write, but also more performance-effective.
+
+Additionally, replacing I/O layers with network interfaces naturally extends the system into distributed computing.
+
+## Signal Friendliness
+
+Although OS-level signals are simple and often underutilized for concurrency,  
+Orbital Design enables practical use of signals for multi-worker environments.
+
+This can provide a minor advantage when building cooperative worker pools.
+
+## Compatibility with Systemd
+
+Systemd's `@.service` unit files support multi-instance execution.
+
+Agents designed with Orbital Design require no more than a worker name as an argument, making them trivially scalable to multiple instances.  
+This provides a low-effort pathway to multi-worker deployments, with restarts handled by Systemd itself.
+
+## Compatibility with Job Schedulers
+
+Orbital Design is not limited to multi-worker or multi-instance models.  
+It is especially well-suited to systems that rely on periodic execution by job schedulers.
+
+While worker-driven systems react to runtime state, job schedulers operate on time-based triggers.  
+Thanks to its stateless model, Orbital Design allows agents to run regardless of timing or system condition.
+
+## Shell Script Friendly
+
+Each agent has a clear and narrow scope, with no need for shared database schemas.  
+This makes it easy to write parts of the system in shell scripts where appropriate.
+
+In practice, this results in simpler and more maintainable solutions in more cases than expected.
+
+## Replaceable Components
+
+Programs in Orbital Design are small and well-scoped.  
+When a language, library, or performance characteristic becomes a limitation, swapping out components comes at low cost.
+
+This helps maintain long-term system health and avoids software decay.
+
+# Finally
+
+“The library is minimal. The idea is not.”
 
+Orbital Design is not a framework. It's a way of thinking. It thrives where ideas are shared freely.

data/lib/orbitalqueue.rb

@@ -14,19 +14,29 @@ class OrbitalQueue
   class QueueUnexisting < QueueError
   end
 
+  class ItemDestructed < QueueError
+  end
+
+  # Return deferred item to queue
+  def self.resume dir
+    self.new(dir).resume
+  end
+
   # Create queue master in presented dir.
   #
-  # dir: Queue directory
-  # create: If true is given, creates the queue directory when it is missing
+  # dir:: Queue directory
+  # create:: If true is given, creates the queue directory when it is missing
   def initialize dir, create=false
     @queue_dir = dir
 
-    unless File.exist?(File.join(dir, ".checkout"))
-      if create
-        require 'fileutils'
-        FileUtils.mkdir_p(File.join(dir, ".checkout"))
-      else
-        raise QueueUnexisting.new("Queue directory #{dir} does not exist.")
+    %w:.checkout .defer .retry .archive:.each do |subdir|
+      unless File.exist?(File.join(dir, subdir))
+        if create
+          require 'fileutils'
+          FileUtils.mkdir_p(File.join(dir, subdir))
+        else
+          raise QueueUnexisting.new("Queue directory #{dir} does not exist.")
+        end
       end
     end
   end
@@ -45,6 +55,12 @@ class OrbitalQueue
   
   # Pop data from queue.
   # Popped queue items are placed in the checkout directory. After processing is complete, +#complete+ must be called to remove the item from the queue.
+  #
+  # If block is given, complete automatically after yield.
+  #
+  # :call-seq:
+  #   pop()               -> queue_object
+  #   pop() {|data| ... } -> queue_id
   def pop
     queue_data = nil
     queue_id = nil
@@ -57,38 +73,225 @@ class OrbitalQueue
         next
       end
 
-      queue_data = Marshal.load(File.read File.join(@queue_dir, ".checkout", File.basename(qf)))
+      data = Marshal.load(File.read File.join(@queue_dir, ".checkout", File.basename(qf)))
       queue_id = File.basename qf, ".marshal"
 
-      if Hash === queue_data
-        queue_data[:queue_id] = queue_id
-      else
-        queue_data = {queue_id: queue_id, data: queue_data}
-      end
+      queue_data = OrbitalQueue::QueueObject.new(self, data, queue_id)
       break
     end
 
-    queue_data
+    if queue_data && block_given?
+      yield queue_data.data
+      complete queue_id
+    else
+      queue_data
+    end
   end
 
-  # Pop data from queue and remove it from queue.
+  # Pop data and remove it from queue.
+  #
+  # :call-seq:
+  #   pop!() -> queue_object
   def pop!
-    queue_data = pop
-    if queue_data
-      complete queue_data[:queue_id]
+    queue_item = pop
+    if queue_item
+      queue_item.complete
+    end
+
+    queue_item
+  end
+
+  # Iterate each queue item data.
+  def each
+    while item = pop
+      yield item.data
+      item.complete
     end
+  end
 
-    queue_data
+  # Iterate each queue item.
+  def each_item
+    while item = pop
+      yield item
+      item.complete unless item.deferred?
+    end
   end
 
   # Remove checked out queue item.
   def complete queue_id
     begin
-      File.delete(File.join(@queue_dir, ".checkout", (queue_id + ".marshal")))
+      checkout_file = File.join(@queue_dir, ".checkout", (queue_id + ".marshal"))
+      retry_file = File.join(@queue_dir, ".retry", (queue_id + ".marshal"))
+      File.delete(checkout_file)
+      File.delete(retry_file) if File.exist?(retry_file)
     rescue SystemCallError => e
       raise QueueRemoveError, "Failed to complete queue #{queue_id}: #{e.class}"
     end
 
     queue_id
   end
+
+  # Delete all related files with queue_id, and raise ItemDectructed exception.
+  def destruct queue_id
+    queue_files = Dir.glob([@queue_dir, "**", (queue_id + ".marshal")].join("/"), File::FNM_DOTMATCH)
+    File.delete(*queue_files) unless queue_files.empty?
+    raise ItemDestructed, "#{queue_id} is destructed."
+  end
+
+  # Archive current queue relative data and call +destruct+.
+  # This method should be called from QueueObject.
+  def archive queue_id, data, archiveinfo_additional={} # :nodoc:
+    archiveinfo = archiveinfo_additional.merge({
+      archived_at: Time.now.to_i
+    })
+
+    retry_data = load_retryobj queue_id
+
+    archive_data = {
+      archiveinfo: archiveinfo,
+      retry_data: retry_data,
+      data: data
+    }
+
+    File.open(File.join(@queue_dir, ".archive", (["archive", archiveinfo[:archived_at], queue_id].join("-") + ".marshal")), "w") {|f| Marshal.dump archive_data, f}
+
+    destruct queue_id
+  end
+
+  # Mark queue item as deferred.
+  # 
+  # :call-seq:
+  #   defer(queue_id, time_at, max_count=nil) -> retry_data | nil
+  #   defer() {|retry_data| ... }             -> retry_data | nil
+  def defer queue_id, time_at=nil, max_count=nil
+    retry_data = load_retryobj queue_id
+    retry_data[:count] += 1
+    if block_given?
+      yield retry_data
+    else
+      unless time_at
+        raise ArgumentError, "time_at is required when no block is given."
+      end
+
+      if max_count && retry_data[:count] > max_count
+        destruct queue_id
+      end
+      retry_data[:until] = time_at.to_i
+    end
+
+    dump_retryobj queue_id, retry_data
+
+    checkout_path = File.join(@queue_dir, ".checkout", (queue_id) + ".marshal")
+    defer_path = File.join(@queue_dir, ".defer", (queue_id) + ".marshal")
+    File.rename checkout_path, defer_path
+
+    retry_data
+  rescue ItemDestructed
+    nil
+  end
+
+  # Return deferred item to queue.
+  def resume
+    now = Time.now.to_i
+    deferred_files = Dir.children(File.join(@queue_dir, ".retry"))
+    deferred_files.each do |fn|
+      retry_path = File.join(@queue_dir, ".retry", fn)
+      retry_data = Marshal.load File.read retry_path
+
+      if retry_data[:until] < now
+        queue_path = File.join(@queue_dir, fn)
+        defer_path = File.join(@queue_dir, ".defer", fn)
+        File.rename(defer_path, queue_path)
+      end
+    end
+
+    nil
+  end
+
+  private
+
+  # Save to .retry
+  def dump_retryobj queue_id, data
+    retry_path = File.join(@queue_dir, ".retry", (queue_id) + ".marshal")
+    File.open(retry_path, "w") {|f| Marshal.dump data, f }
+    nil
+  end
+
+  # Load from .retry
+  def load_retryobj queue_id
+    retry_path = File.join(@queue_dir, ".retry", (queue_id) + ".marshal")
+    retry_data = nil
+    if File.exist? retry_path
+      retry_data = Marshal.load File.read retry_path
+    else
+      retry_data = {
+        count: 0,
+        until: nil
+      }
+    end
+
+    retry_data
+  end
 end
+
+# Queue item capsule.
+class OrbitalQueue::QueueObject
+  def initialize(queue, data, queue_id)
+    @queue = queue
+    @data = data
+    @queue_id = queue_id
+    @completed = false
+    @deferred = false
+  end
+
+  attr_reader :data
+
+  # Another complete interface.
+  def complete
+    if @completed
+      nil
+    else
+      @queue.complete(@queue_id)
+      @completed = true
+      @queue_id
+    end
+  end
+
+  # Wrap for the end of queue item.
+  def destruct
+    @completed = true
+    @queue.destruct(@queue_id)
+  end
+
+  # Archive current queue relative data and call +destruct+.
+  def archive archiveinfo_additional={}
+    @completed = true
+    @queue.archive @queue_id, @data, archiveinfo_additional
+  end
+
+  # Terrible redundunt method.
+  def complete? # :nodoc:
+    @completed
+  end
+
+  # Retry later.
+  #
+  # time_at:: Deferring retry until this time
+  # max_count:: Retry count limit
+  #
+  # :call-seq:
+  #   defer(time_at, max_count=nil) -> retry_data
+  #   defer() {|retry_data| ... }   -> retry_data
+  def defer time_at=nil, max_count=nil, &block
+    if block
+      @queue.defer(@queue_id, &block)
+    else
+      @queue.defer(@queue_id, time_at, max_count)
+    end
+    @deferred = true
+  end
+
+  def deferred?
+    @deferred
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: orbitalqueue
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.0.3
 platform: ruby
 authors:
 - Masaki Haruka