orbitalqueue 0.0.2 → 0.0.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d331ea29dd6bc80aa3c1127749f782845d1f23c2a9729f987af9128f1922bf21
-  data.tar.gz: 5f7c04a0cef45399d4726900330ac96b1714e3275545a2d1e4e209fe61aec6c9
+  metadata.gz: 7eab13f9b8b7353b492bfded166685f5ba54135bc23c561b723ca0ea7e0c8c72
+  data.tar.gz: 8d6b59b0a2448afed4df5fda116ae31352bd4289c9f47290d91bfdcb99d9bebd
 SHA512:
-  metadata.gz: a7d5ead3e5494a3f38808e9b363b1ceb624168e294ee6a7165630092ec3ff3a58448234f434411f865539e89d8aa5f1cdf3100adde94ca6cc26066ec5c1d0795
-  data.tar.gz: 7aab8b14af6f147b0b7276e37baee3db4324ca4052445c1fefcdd094f47e64e770f0a3357eec8a69d35ebbedd2263b7b3b5d208e457c73b5f6b3f34a5560ba39
+  metadata.gz: 8499346b48cc77f21035d529d1119ae0ad5a56efc99b6823890b642776925adf65b6768104589262bd5e5c80d542fc3b2dee9d716e0ba888e22799a379932444
+  data.tar.gz: 1d58db0f106088bd71ad7df70ebe537cbef6b699642cac5c73d0bfcadbac3906a7a478946f98ad2e641554850e4aa00b6a9289ebeadaee02b1e0089f6395c703

data/README.md

@@ -49,15 +49,274 @@ item.complete
 
 Calling `#pop` retrieves a single item from the queue in no particular order.
 
+```ruby
+queue = OrbitalQueue.new("/path/to/queue")
+item = queue.pop!
+
+# Something, something...
+```
+
 The retrieved item enters a checkout state, and must be finalized by calling `#complete` once processing is finished.
 
 If guaranteed completion is not required and the item should be removed immediately upon retrieval, use `#pop!` instead.
 
-Both `#pop` and `#pop!` return a queue item as a `Hash`. If the original object was not a `Hash`, it will be stored under the `:data` key. Regardless of the original type, the queue ID is always stored under the `:queue_id` key.
+Both `#pop` and `#pop!` methods returns `OrbitalQueue::QueueObject` object.
+You can access queue data via `OrbitalQueue::QueueObject#data`.
+
+When calling `#complete`, `#destruct`, or `#defer` directly on `OrbitalQueue`, `queue_id` must be given.  
+If you use these methods via `OrbitalQueue::QueueObject`, the `queue_id` is internally handled and can be omitted.
+
+### Dequeue with block
+
+You can call `#pop` with a block.
+
+When call with block, block is called with queue data as an argument.
+
+queue item automatically complete when blocke ends without error.
+
+The `#pop` method can be called with an optional block.  
+When used this way, the block is invoked with the item's data.  
+After successful execution (without exceptions), the item is considered complete and removed from the queue.
+
+```ruby
+queue = OrbitalQueue.new("/path/to/queue")
+queue.pop do |data|
+  #...
+end
+```
+
+### Dequeue loop
+
+While technically possible to loop over `#pop`, it doesn't support block-based iteration.
+
+Use `#each` when you want to process items with a block in a loop.  
+It provides a clean and idiomatic Ruby style for sequential processing.
+
+For full control over each item, use `#each_item`, which yields a `QueueObject` instead of just the data.
+
+`#each_item` iterates over queue items as `QueueObject` instances, not raw data.  
+This allows direct control over each job—for example, deferring its execution using `#defer`.
+
+Unlike `#each`, which automatically marks items as complete after the block runs,  
+`#each_item` exposes queue control for cases where completion isn't guaranteed or deferred handling is needed.
+
+### Job deferral
+
+`OrbitalQueue` supports job deferral, enabling queue items to be scheduled for retry or postponed execution with precise control.
+
+Calling `#defer` transitions a queue item into the deferred state.  
+This moves the item's file into the `.defer` directory and creates a retry metadata file in `.retry`.
+
+Once an item has been deferred, it is associated with retry information, referred to as `retry_data`.  
+This is a `Hash` object that tracks rescheduling behavior.
+
+`#defer` uses `retry_data` to determine whether the item can be retried, including retry count limits.  
+When called with a block, `#defer` yields `retry_data` as an argument, allowing custom modifications.
+
+Regardless of how it's called, `retry_data` is persisted after the method returns.  
+To control deferral behavior, modify values inside `retry_data`—typically by changing the `:until` field (Unix timestamp).  
+This field specifies when the item should become eligible for re-queueing, making it ideal for implementing backoff strategies.
+
+```ruby
+queue.each_item do |item|
+  begin
+    # Something...
+  rescue
+    item.defer(Time.now + 300)     # Retry after 5 minutes.
+  end
+end
+```
+
+with block:
+
+```ruby
+queue.each_item do |item|
+  begin
+    # Something...
+  rescue
+    item.defer do |retry_item|
+      if retry_item[:count] > 5
+        item.destruct
+      else
+        retry_item[:until] = Time.now + 300 * retry_item[:count]
+      end
+    end
+  end
+end
+```
+
+The `#defer` method's main role is to move the queue item file into the `.defer` directory.  
+Since `OrbitalQueue` operates without a server, it cannot scan `.defer` efficiently or restore deferred items automatically.
+
+Aside from the internal keys `:count` and `:until`, all other values in `retry_data` are preserved as-is.  
+You can freely store custom metadata inside it—such as failure reasons or backoff parameters.
+
+Note: `defer` block is called after `:count` is incremented.
+
+The `#destruct` method removes all files associated with a queue item and raises `OrbitalQueue::ItemDestructed`.  
+This exception is caught inside `#defer`, allowing `#destruct` to abort the entire deferral process.
+
+⚠️ Do not rescue `OrbitalQueue::ItemDestructed` within a `#defer` block.  
+If the block completes normally after destruction, queue integrity may be violated.
+
+The `#archive` method creates a Marshal-serialized file under `.archive`, containing the original data, its `retry_data`, and an `archiveinfo` hash.  
+After archiving, it calls `#destruct` to remove the live queue item.
+
+Archived files are never accessed by OrbitalQueue itself.
+
+Note: Because `archive` discards in-memory `retry_data`, you cannot modify it before archiving.  
+Instead, extra metadata should be passed as arguments to `archive` and will be merged into `archiveinfo`.
+
+```ruby
+queue.each_item do |item|
+  begin
+    #...
+  rescue
+    item.defer do |retry_data|
+      if retry_data[:count] > 5
+        item.archive({reason: "Host timeout"})
+      end
+    end
+  end
+end
+```
+
+### Resume deferred job
+
+Deferred queue items must be manually restored using the `resume` method.  
+This method is typically executed by a separate worker from the one handling regular queue operations.
+
+It is defined as an instance method:
+
+```ruby
+queue = OrbitalQueue.new("/path/to/queue")
+queue.resume
+```
+
+For convenience, resume can also be called as a class method:
+
+```ruby
+OrbitalQueue.resume("/path/to/queue")
+```
 
 # About Orbital Design
 
 ## Description
 
-## Design pattern rules
+Orbital Design is a programming pattern optimized for distributed systems.  
+It is especially well-suited to environments where:
+
+* New data constantly arrives without pause
+* Processing workloads vary in complexity and demand asymmetric distribution
+* Systems start small but must scale seamlessly to clustered deployments
+
+## Philosophy
+
+Orbital Design distinguishes between "agents" and "workers".  
+In most cases, an agent refers to a program, while a worker is a process.
+
+The core principle is that **workers only need to care about what they do**.  
+Upon starting, a worker picks a single available job prepared for it and executes it—no coordination or negotiation required.
+
+This behavior mirrors that of individuals in a larger society, or cells within a living organism.  
+Each unit performs its specific role independently.
+
+This philosophy is deeply aligned with the Unix principle:  
+_"Do one thing, and do it well."_
+
+## Core Rules of Orbital Design
+
+Orbital Design defines a set of principles to preserve decoupling, clarity, and safety in distributed systems:
+
+- *Agents must remain small and focused*.  
+  Each agent is responsible for doing one thing, and doing it well.
+
+- *Workers must not access data unrelated to their task*, nor inspect other workers' state or progress.
+
+- *Write access to a database or dataset must be held by exactly one agent*.  
+  This prevents conflicting updates and maintains integrity.
+
+- *Deletion from a database may only be performed by:*
+  - A worker with exclusive read access to the data, or
+  - A sweeper worker that receives notifications from all readers
+
+- *Agents must not block on I/O*.  
+  Blocking input/output disrupts concurrency and undermines distributed fairness.
+
+## Benefits of Orbital Design
+
+### Ease of Implementation
+
+Each program is small and focused, with clearly defined responsibilities.  
+Because agents cannot access global state and avoid blocking operations, race conditions are structurally prevented.
+
+This allows each unit to concentrate solely on its task—no need to worry about concurrency or system state.
+
+## Simplicity
+
+Orbital Design requires minimal complexity.  
+It does not depend on heavy frameworks or advanced techniques.
+
+It can be fully implemented using standard OS features such as file systems, processes, and signals.  
+No special measures are needed to achieve scalability.
+
+## Language Agnosticism
+
+Programs are isolated and do not interfere with one another.  
+This allows you to implement each agent in any language that suits the task.
+
+You can choose a language based on convenience, libraries, or performance.
+Critical paths can be written in C, C++, Rust, or Nim as needed, while simpler agents may use scripting languages.
+
+Even agents with similar functionality can be written in different languages depending on input format or operational context.
+
+## Parallelism and Decomposition
+
+Restricted I/O paths eliminate contention during parallel execution.  
+By following the design pattern, concurrency becomes straightforward.
+
+No locking or synchronization is required—so parallel processing not only becomes easier to write, but also more performance-effective.
+
+Additionally, replacing I/O layers with network interfaces naturally extends the system into distributed computing.
+
+## Signal Friendliness
+
+Although OS-level signals are simple and often underutilized for concurrency,  
+Orbital Design enables practical use of signals for multi-worker environments.
+
+This can provide a minor advantage when building cooperative worker pools.
+
+## Compatibility with Systemd
+
+Systemd's `@.service` unit files support multi-instance execution.
+
+Agents designed with Orbital Design require no more than a worker name as an argument, making them trivially scalable to multiple instances.  
+This provides a low-effort pathway to multi-worker deployments, with restarts handled by Systemd itself.
+
+## Compatibility with Job Schedulers
+
+Orbital Design is not limited to multi-worker or multi-instance models.  
+It is especially well-suited to systems that rely on periodic execution by job schedulers.
+
+While worker-driven systems react to runtime state, job schedulers operate on time-based triggers.  
+Thanks to its stateless model, Orbital Design allows agents to run regardless of timing or system condition.
+
+## Shell Script Friendly
+
+Each agent has a clear and narrow scope, with no need for shared database schemas.  
+This makes it easy to write parts of the system in shell scripts where appropriate.
+
+In practice, this results in simpler and more maintainable solutions in more cases than expected.
+
+## Replaceable Components
+
+Programs in Orbital Design are small and well-scoped.  
+When a language, library, or performance characteristic becomes a limitation, swapping out components comes at low cost.
+
+This helps maintain long-term system health and avoids software decay.
+
+# Finally
+
+“The library is minimal. The idea is not.”
 
+Orbital Design is not a framework. It's a way of thinking. It thrives where ideas are shared freely.

data/lib/orbitalqueue.rb

@@ -14,19 +14,29 @@ class OrbitalQueue
   class QueueUnexisting < QueueError
   end
 
+  class ItemDestructed < QueueError
+  end
+
+  # Return deferred item to queue
+  def self.resume dir
+    self.new(dir).resume
+  end
+
   # Create queue master in presented dir.
   #
-  # dir: Queue directory
-  # create: If true is given, creates the queue directory when it is missing
+  # dir:: Queue directory
+  # create:: If true is given, creates the queue directory when it is missing
   def initialize dir, create=false
     @queue_dir = dir
 
-    unless File.exist?(File.join(dir, ".checkout"))
-      if create
-        require 'fileutils'
-        FileUtils.mkdir_p(File.join(dir, ".checkout"))
-      else
-        raise QueueUnexisting.new("Queue directory #{dir} does not exist.")
+    %w:.checkout .defer .retry .archive:.each do |subdir|
+      unless File.exist?(File.join(dir, subdir))
+        if create
+          require 'fileutils'
+          FileUtils.mkdir_p(File.join(dir, subdir))
+        else
+          raise QueueUnexisting.new("Queue directory #{dir} does not exist.")
+        end
       end
     end
   end
@@ -47,6 +57,10 @@ class OrbitalQueue
   # Popped queue items are placed in the checkout directory. After processing is complete, +#complete+ must be called to remove the item from the queue.
   #
   # If block is given, complete automatically after yield.
+  #
+  # :call-seq:
+  #   pop()               -> queue_object
+  #   pop() {|data| ... } -> queue_id
   def pop
     queue_data = nil
     queue_id = nil
@@ -66,7 +80,7 @@ class OrbitalQueue
       break
     end
 
-    if block_given?
+    if queue_data && block_given?
       yield queue_data.data
       complete queue_id
     else
@@ -74,7 +88,10 @@ class OrbitalQueue
     end
   end
 
-  # Pop data from queue and remove it from queue.
+  # Pop data and remove it from queue.
+  #
+  # :call-seq:
+  #   pop!() -> queue_object
   def pop!
     queue_item = pop
     if queue_item
@@ -84,16 +101,138 @@ class OrbitalQueue
     queue_item
   end
 
+  # Iterate each queue item data.
+  def each
+    while item = pop
+      yield item.data
+      item.complete
+    end
+  end
+
+  # Iterate each queue item.
+  def each_item
+    while item = pop
+      yield item
+      item.complete unless item.deferred?
+    end
+  end
+
   # Remove checked out queue item.
   def complete queue_id
     begin
-      File.delete(File.join(@queue_dir, ".checkout", (queue_id + ".marshal")))
+      checkout_file = File.join(@queue_dir, ".checkout", (queue_id + ".marshal"))
+      retry_file = File.join(@queue_dir, ".retry", (queue_id + ".marshal"))
+      File.delete(checkout_file)
+      File.delete(retry_file) if File.exist?(retry_file)
     rescue SystemCallError => e
       raise QueueRemoveError, "Failed to complete queue #{queue_id}: #{e.class}"
     end
 
     queue_id
   end
+
+  # Delete all related files with queue_id, and raise ItemDectructed exception.
+  def destruct queue_id
+    queue_files = Dir.glob([@queue_dir, "**", (queue_id + ".marshal")].join("/"), File::FNM_DOTMATCH)
+    File.delete(*queue_files) unless queue_files.empty?
+    raise ItemDestructed, "#{queue_id} is destructed."
+  end
+
+  # Archive current queue relative data and call +destruct+.
+  # This method should be called from QueueObject.
+  def archive queue_id, data, archiveinfo_additional={} # :nodoc:
+    archiveinfo = archiveinfo_additional.merge({
+      archived_at: Time.now.to_i
+    })
+
+    retry_data = load_retryobj queue_id
+
+    archive_data = {
+      archiveinfo: archiveinfo,
+      retry_data: retry_data,
+      data: data
+    }
+
+    File.open(File.join(@queue_dir, ".archive", (["archive", archiveinfo[:archived_at], queue_id].join("-") + ".marshal")), "w") {|f| Marshal.dump archive_data, f}
+
+    destruct queue_id
+  end
+
+  # Mark queue item as deferred.
+  # 
+  # :call-seq:
+  #   defer(queue_id, time_at, max_count=nil) -> retry_data | nil
+  #   defer() {|retry_data| ... }             -> retry_data | nil
+  def defer queue_id, time_at=nil, max_count=nil
+    retry_data = load_retryobj queue_id
+    retry_data[:count] += 1
+    if block_given?
+      yield retry_data
+      retry_data[:until] = retry_data[:until].to_i
+    else
+      unless time_at
+        raise ArgumentError, "time_at is required when no block is given."
+      end
+
+      if max_count && retry_data[:count] > max_count
+        destruct queue_id
+      end
+      retry_data[:until] = time_at.to_i
+    end
+
+    dump_retryobj queue_id, retry_data
+
+    checkout_path = File.join(@queue_dir, ".checkout", (queue_id) + ".marshal")
+    defer_path = File.join(@queue_dir, ".defer", (queue_id) + ".marshal")
+    File.rename checkout_path, defer_path
+
+    retry_data
+  rescue ItemDestructed
+    nil
+  end
+
+  # Return deferred item to queue.
+  def resume
+    now = Time.now.to_i
+    deferred_files = Dir.children(File.join(@queue_dir, ".retry"))
+    deferred_files.each do |fn|
+      retry_path = File.join(@queue_dir, ".retry", fn)
+      retry_data = Marshal.load File.read retry_path
+
+      if retry_data[:until] < now
+        queue_path = File.join(@queue_dir, fn)
+        defer_path = File.join(@queue_dir, ".defer", fn)
+        File.rename(defer_path, queue_path)
+      end
+    end
+
+    nil
+  end
+
+  private
+
+  # Save to .retry
+  def dump_retryobj queue_id, data
+    retry_path = File.join(@queue_dir, ".retry", (queue_id) + ".marshal")
+    File.open(retry_path, "w") {|f| Marshal.dump data, f }
+    nil
+  end
+
+  # Load from .retry
+  def load_retryobj queue_id
+    retry_path = File.join(@queue_dir, ".retry", (queue_id) + ".marshal")
+    retry_data = nil
+    if File.exist? retry_path
+      retry_data = Marshal.load File.read retry_path
+    else
+      retry_data = {
+        count: 0,
+        until: nil
+      }
+    end
+
+    retry_data
+  end
 end
 
 # Queue item capsule.
@@ -103,6 +242,7 @@ class OrbitalQueue::QueueObject
     @data = data
     @queue_id = queue_id
     @completed = false
+    @deferred = false
   end
 
   attr_reader :data
@@ -118,8 +258,41 @@ class OrbitalQueue::QueueObject
     end
   end
 
+  # Wrap for the end of queue item.
+  def destruct
+    @completed = true
+    @queue.destruct(@queue_id)
+  end
+
+  # Archive current queue relative data and call +destruct+.
+  def archive archiveinfo_additional={}
+    @completed = true
+    @queue.archive @queue_id, @data, archiveinfo_additional
+  end
+
   # Terrible redundunt method.
-  def complete?
+  def complete? # :nodoc:
     @completed
   end
-end
+
+  # Retry later.
+  #
+  # time_at:: Deferring retry until this time
+  # max_count:: Retry count limit
+  #
+  # :call-seq:
+  #   defer(time_at, max_count=nil) -> retry_data
+  #   defer() {|retry_data| ... }   -> retry_data
+  def defer time_at=nil, max_count=nil, &block
+    if block
+      @queue.defer(@queue_id, &block)
+    else
+      @queue.defer(@queue_id, time_at, max_count)
+    end
+    @deferred = true
+  end
+
+  def deferred?
+    @deferred
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: orbitalqueue
 version: !ruby/object:Gem::Version
-  version: 0.0.2
+  version: 0.0.4
 platform: ruby
 authors:
 - Masaki Haruka