excadg 0.1.4 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1081c1531553e5ca9113aab9d0f8dfaa7982438438f646df74fded7a17ec0e99
-  data.tar.gz: 88d4ecf74086eddcea280c44844c47097aeca252b6f9170f585f0516368928f0
+  metadata.gz: 49311f7d62494f4eea817789e886391472cb5a4b9fc971e6d179baf9cbb679b6
+  data.tar.gz: 7d36c94f2b7c13441bd36c4d44c9a1a0e77e1b96cb14fa2ace12cf8068ff3668
 SHA512:
-  metadata.gz: a086d015c957d0b86ecf3ec8de789aa4f151ea1687c6ddde4f5454b28ea1f4737bafc1f81ccac9dd0a1fed45603c6fae8dfc71649dcbe383f8cb9baf3f887e70
-  data.tar.gz: 394abc39d249922bd7eec046e748673e0ad7eb294f3c91db5a15bbea26a99486f159a55411f1baed09c80f6e3e919654bce87eb97a441461c9b97e464ee97b96
+  metadata.gz: 5586527ad35c02823db4f521cb74f34a819e9164d538e8611b3cb68d3fe4c0b8b8c983c5fd91e72ff5cc327fc3d0d0c98ff04d18a4c853e43e26f04d95f742bd
+  data.tar.gz: a5a1f2cd5b58e8319ceb64794848006898a76cb9ba0ffea79be6ad4bd4d7fd0bd5ff730dc3aaae7437ebf09f0fdcef64a5466c48d635881e0dabb3cd02f2263e

data/README.md

@@ -21,7 +21,7 @@ Main usage pattern for this project is a framework. It implies that you:
 ### Payload implementation
 
 Payload is a piece of code to execute in a single vertex. This code should be self-contained and use specific ways to communicate with other vertices.  
-There is a [{ExcADG::Payload}](lib/excadg/payload.rb) module to make a custom payload class. Read through its documentation to get familiar with its interface.  
+There is a [{ExcADG::Payload}](lib/excadg/payload.rb) module to make a custom payload class. Read through its documentation to get familiar with the interface.  
 
 Here is a very basic example of a payload that echoes a message:
 
@@ -33,29 +33,31 @@ class MyPayload
   end
 end
 ```
-More payload examples could be found in [{ExcADG::Payload::Example}](lib/excadg/payload/example.rb).
+More payload examples could be found in the [{ExcADG::Payload::Example}](lib/excadg/payload/example.rb) module.
 
 Any payload has 3 parts:
 - code
 - arguments
 - dependencies data
 
-**Code** is defined in {Payload#get} method. E.g. `system 'echo here I am'` in the above example.  
-**Arguments** are defined during payload object contruction (see {Payload#initialize}) as `args:` parameter. E.g. `MyPayload.new args: 'my custom message'`.  
+**Code** is defined in {ExcADG::Payload#get} method. E.g. `system 'echo here I am'` in the above example.  
+**Arguments** could be supplied during payload object contruction (see {ExcADG::Payload#initialize}) as `args:` parameter. E.g. `MyPayload.new args: 'my custom message'`.  
 **Dependencies data** is provided by the framework and has all [{ExcADG::VStateData}](lib/excadg/vstate_data.rb) returned by the dependencies.
 
 ### Vertices constructing
 
-Vertex is a execution graph's building block. It executes its payload when all its dependencies are finished.  
-Vertex can be created this way: `Vertex.new name: :myvertex, payload: MyPayload.new`. This vertex doesn't have any dependencies, so it starts execution immediately upon construction. Vertices without dependencies are always aneeded in the graph to start it.  
+Vertex is an execution graph's building block. It executes its payload when all its dependencies are finished.  
+Vertex can be created this way: `Vertex.new name: :myvertex, payload: MyPayload.new`. This vertex doesn't have any dependencies, so it starts execution immediately upon construction. Vertices without dependencies are always needed in the graph to start it.  
 
-Another "kind" of vertices are vertices with dependencies. Dependencies are other vertices that the vertex waits to finish successfully before running its own payload.  
-> Example: `Vertex.new name: :other_vertex, payload: MyPayload.new, deps: [:myvertex]` has exactly 1 dependency - vertex called `:myvertex`  
-> `:other_vertex` won't start until `:myvertex` finishes.  
-> Moreover, as described in [payload](#payload), `:other_vertex`'s payload will have access to data returned by `:myvertex`'s payload.  
-> In this case, it'd be what `system 'echo here I am'` returns - `true`.
+Another "kind" of vertices are vertices with dependencies. Dependencies are an array other vertices that the vertex waits to finish successfully before running its own payload.  
 
-Dependencies could be specified both - using {ExcADG::Vertex} objects and names. E.g.
+Example:  
+`Vertex.new name: :other_vertex, payload: MyPayload.new, deps: [:myvertex]` has exactly 1 dependency - the vertex called `:myvertex`  
+`:other_vertex` won't start until `:myvertex` finishes.  
+Moreover, as described in [payload](#excadgpayload), `:other_vertex`'s payload will have access to data returned by `:myvertex`'s payload.  
+In this case, it'd be what `system 'echo here I am'` returns - `true`.
+
+Dependencies could be specified with both - {ExcADG::Vertex} objects and names. E.g.
 ``` ruby
 Broker.run
 
@@ -67,27 +69,27 @@ Vertex.new name: :final, payload: MyPayload.new, deps: [v1, :v2]
 Broker.wait_all
 ```
 
-*See [Broker section](#broker) for `Broker.run` and `Broker.wait_all` usage.*
+*See [Broker section](#excadgbroker) for `Broker.run` and `Broker.wait_all` usage.*
 
 Using actual objects looks simpler, but it's less convenient, as it requires you to construct all vertices as they appear in the execution graph.  
-However, names allows you to spawn vertices in arbitrary order and expect framework to figure execution order on the fly. See [run tool](#tool) as an example of using names.
+However, names allows you to spawn vertices in arbitrary order and expect framework to figure execution order on the fly. See [run tool](#tool)'s code as an example of using names.
 
-*There is no need to store {ExcADG::Vertex} objects, as it and its data are available through [Broker](#broker)'s [DataStore](#data-store) and there is no interface to communicate with a {ExcADG::Vertex} directly.*
+*There is no need to store {ExcADG::Vertex} objects, as it and its data are available through [Broker](#excadgbroker)'s [DataStore](#excadgdatastore) and there is no interface to communicate with an {ExcADG::Vertex} directly.*
 
 # Internals
 
 ## Overview
 
 This framework allows to spawn vertices. Once created, a vertex with payload starts immediately.
-This means that there is no central place where execution seqence is controlled besides vertex's own mechanism to wait for dependencies.
+This means that there is no central place that controls execution seqence besides vertex's own mechanism to wait for dependencies.
 
-As Ractors doesn't allow to reliably exchange data between each other at the moment, the main Ractor (thread) has to spawn a broker to synchronize data exchange. See [broker](#broker) sections to learn more.
+As Ractors doesn't allow to reliably exchange data between each other at the moment, the main Ractor (thread) has to spawn a broker to synchronize data exchange. See [broker](#excadgbroker) sections to learn more.
 
 This framework is based on [Ractors](https://docs.ruby-lang.org/en/master/ractor_md.html). It could be useful to get familiar with ractors before reading further.
 
 ## Vertice processing states
 
-Internally, each vertice goes through a sequence of states. Now it's **new**, **ready**, **done** and **failed**. Stages are controlled by the [{ExcADG::StateMachine}](#statemachine).
+Internally, each vertice go through a sequence of states. Now it's **new**, **ready**, **done** and **failed**. Stages are controlled by the [{ExcADG::StateMachine}](#excadgstatemachine).
 
 {ExcADG::Vertex} starts as a **new** vertex and waits for its dependencies.  
 When vertex received all dependencies states and made sure they're **done**, it becomes **ready** and starts its payload.  
@@ -103,13 +105,13 @@ When a vertex changes its state, it (actually, state machine does that) notifies
 Broker is desired to be as thin as possible to keep most of the work for vertices.
 
 Each application should invoke {Broker.run} to enable messages processing.  
-Its counterpart is {Broker.wait_all} which waits for all known vertices to reach one of the terminal states (**done** or **failed**) within specified timeout. Same as `Broker.run`, it spawns a thread and returns it. The main application could keep it in background and query or `.join` on it once all vertices are spawned. Once the thread finishes, the main app could lookup vertices execution results in {Broker.data_store} (see [DataStore](#data-store)).
+Its counterpart is {Broker.wait_all} which waits for all known vertices to reach one of the terminal states (**done** or **failed**) within specified timeout. Same as `Broker.run`, it spawns a thread and returns it. The main application could keep it in background and query or `.join` on it once all vertices are spawned. Once the thread finishes, the main app could lookup vertices execution results in {Broker.data_store} (see [DataStore](#excadgdatastore)).
 
 > Beware that broker constantly uses main Ractor's ports - incoming and outgoing. Hence, `Ractor#take`, `Ractor.yield` or any other messaging in the main ractor conflict with broker.
 
 ## {ExcADG::DataStore}
 
-It's a mostly internal [Broker's](#broker) object that holds all vertice's [{ExcADG::VStateData}](lib/excadg/vstate_data.rb).
+It's a mostly internal [Broker's](#excadgbroker) object that holds all vertice's [{ExcADG::VStateData}](lib/excadg/vstate_data.rb).
 
 ## {ExcADG::StateMachine}
 
@@ -117,7 +119,7 @@ State machine is a small internal mechanism that helps vertices to
 - do state transitions
 - preserve state transition results locally
 
-State machine has transitions graph that is common for all vertices. Vertices bind certain logic to transitions. E.g. "wait for dependencies" or "run the payload". State transition mechanism ensures to run workload associated to the currently possible transition, process errors (if any) and send results to [Broker](#broker).
+State machine has transitions graph that is common for all vertices. Vertices bind certain logic to transitions. E.g. "wait for dependencies" or "run the payload". State transition mechanism ensures to run workload associated to the currently possible transition, process errors (if any) and send results to [Broker](#excadgbroker).
 
 ## {ExcADG::Payload}
 
@@ -130,7 +132,7 @@ First way is `args` parameter of the {Payload#initialize}. It can be used on pay
 Second way is built-in. Vertex invokes payload with {ExcADG::Array} of dependencies {ExcADG::VStateData}, if payload's `Proc` is parametrized (has arity 1).
 
 > Be mindful about data your payload receives (args) and returns (state data). It could appear incompatible with ractors and cause vertice to fail.  
-> Although these failures are hard to tell beforehand, [state machine](#statemachine) processes them gracefully.
+> Although these failures are hard to tell beforehand, [state machine](#excadgstatemachine) processes them gracefully.
 
 ## {ExcADG::Log}
 
@@ -175,10 +177,6 @@ Logging is disabled by default, but it could be useful to debug tests. Add `ExcA
 
 > here is a list of improvemets could be implementex next
 
-- add timeouts
-  - to payload
-  - to whole vertex
-  - to request processing
 - implement throttling (:suspended state)
   - limit # of running vertices
     - problem: can't find what to suspend

data/lib/excadg/payload/example.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require 'benchmark'
+
 require_relative '../broker'
 require_relative '../request'
 
@@ -55,23 +57,16 @@ module ExcADG
       end
     end
 
-    # calculate Nth member of the function
-    # y = x(n) * (x(n-1)) where x(0) = 0 and x(1) = 1
-    # it illustrates how payload can be customized with @args
-    # and extended by adding methods
-    class Multiplier
+    # payload that occupies a CPU core for several seconds,
+    # is suitable for perfomance tests
+    class Benchmark
       include Payload
       def get
-        -> { get_el x: @args }
-      end
-
-      def get_el x:
-        case x
-        when 0 then 0
-        when 1 then 1
-        else
-          x * get_el(x: x - 1)
-        end
+        lambda {
+          Log.info(::Benchmark.measure {
+            10_000.downto(1) { |i| 10_000.downto(1) { |j| i * j } }
+          })
+        }
       end
     end
 

data/lib/excadg/rtimeout.rb

@@ -9,6 +9,7 @@ module ExcADG
       return block.call if timeout.nil? || timeout.zero?
 
       timed_out = false
+      Thread.report_on_exception = false
       payload = Thread.new { Thread.current[:result] = block.call }
       Thread.new {
         sleep timeout

data/lib/excadg/state_machine.rb

@@ -53,24 +53,30 @@ module ExcADG
       target = target_candidates.first
       Log.debug "found a candidate: #{target}"
       edge = GRAPH.edges.find { |e| e.source == @state && e.target = target }
-      begin
+
+      with_fault_processing {
         @state_transition_data[target] = @state_edge_bindings[edge].call
         @state = target
         Log.debug "moved to #{@state}"
+      }
+      @state_transition_data[@state]
+    end
+
+    # invoke the block provided with state update on failures
+    def with_fault_processing
+      yield if block_given?
+    rescue StandardError => e
+      Log.error "step failed with #{e} / #{e.backtrace}"
+      @state_transition_data[:failed] = e
+      @state = :failed
+    ensure
+      begin
+        Broker.ask Request::Update.new data: state_data
       rescue StandardError => e
-        Log.error "step failed with #{e} / #{e.backtrace}"
         @state_transition_data[:failed] = e
         @state = :failed
-      ensure
-        begin
-          Broker.ask Request::Update.new data: state_data
-        rescue StandardError => e
-          @state_transition_data[:failed] = e
-          @state = :failed
-          Broker.ask Request::Update.new data: state_data
-        end
+        Broker.ask Request::Update.new data: state_data
       end
-      @state_transition_data[@state]
     end
 
     def assert_state_transition_bounds

data/lib/excadg/tui.rb

@@ -12,16 +12,6 @@ module ExcADG
     MAX_VERTEX_TO_SHOW = 10
     DELAY = 0.2
     DEFAULT_BOX_SIZE = { height: 50, width: 150 }.freeze
-    # TODO: do runtime calc
-    BOX_SIZE = {
-      height: DEFAULT_BOX_SIZE[:height] > (IO.console&.winsize&.first || 1000) ? IO.console.winsize.first : DEFAULT_BOX_SIZE[:height],
-      width: DEFAULT_BOX_SIZE[:width] > (IO.console&.winsize&.last || 1000) ? IO.console.winsize.last : DEFAULT_BOX_SIZE[:width]
-    }.freeze
-    CONTENT_SIZE = {
-      height: BOX_SIZE[:height] - 4, # 2 for borders, 1 for \n, 1 for remark
-      width: BOX_SIZE[:width] - 5 # 2 for borders, 2 to indent
-    }.freeze
-    LINE_TEMPLATE = "| %-#{CONTENT_SIZE[:width]}s |\n".freeze
 
     @started_at = DateTime.now.strftime('%Q').to_i
 
@@ -47,20 +37,21 @@ module ExcADG
       # @param content is a list of lines to print
       def print_in_box content
         clear
-        print "+-#{'-' * CONTENT_SIZE[:width]}-+\n"
-        content[..CONTENT_SIZE[:height]].each { |line|
-          if line.size > CONTENT_SIZE[:width]
-            printf LINE_TEMPLATE, "#{line[...(CONTENT_SIZE[:width] - 3)]}..."
+        refresh_sizes
+        print "+-#{'-' * @content_size[:width]}-+\n"
+        content[..@content_size[:height]].each { |line|
+          if line.size > @content_size[:width]
+            printf @line_template, "#{line[...(@content_size[:width] - 3)]}..."
           else
-            printf LINE_TEMPLATE, line
+            printf @line_template, line
           end
         }
-        if content.size < CONTENT_SIZE[:height]
-          (CONTENT_SIZE[:height] - content.size).times { printf LINE_TEMPLATE, ' ' }
+        if content.size < @content_size[:height]
+          (@content_size[:height] - content.size).times { printf @line_template, ' ' }
         else
-          printf LINE_TEMPLATE, '<some content did not fit and was cropped>'[..CONTENT_SIZE[:width]]
+          printf @line_template, '<some content did not fit and was cropped>'[..@content_size[:width]]
         end
-        print "+-#{'-' * CONTENT_SIZE[:width]}-+\n"
+        print "+-#{'-' * @content_size[:width]}-+\n"
       end
 
       def print_summary has_failed, timed_out
@@ -81,6 +72,18 @@ module ExcADG
         print "\e[2J\e[f"
       end
 
+      def refresh_sizes
+        box_size = {
+          height: IO.console&.winsize&.first.nil? || DEFAULT_BOX_SIZE[:height] < IO.console.winsize.first ? DEFAULT_BOX_SIZE[:height] : IO.console.winsize.first,
+          width: IO.console&.winsize&.last.nil? || DEFAULT_BOX_SIZE[:width] < IO.console&.winsize&.last ? DEFAULT_BOX_SIZE[:width] : IO.console.winsize.last
+        }.freeze
+        @content_size = {
+          height: box_size[:height] - 4, # 2 for borders, 1 for \n, 1 for remark
+          width: box_size[:width] - 5 # 2 for borders, 2 to indent
+        }.freeze
+        @line_template = "| %-#{@content_size[:width]}s |\n"
+      end
+
       # make states summary, one for a line with consistent placing
       def state_stats
         skeleton = StateMachine::GRAPH.vertices.collect { |v| [v, []] }.to_h

data/lib/excadg/vertex.rb

@@ -7,6 +7,7 @@ require_relative 'payload'
 require_relative 'request'
 require_relative 'rtimeout'
 require_relative 'state_machine'
+require_relative 'vtimeout'
 
 module ExcADG
   # Individual vertex of the execution graph to run in a separated Ractor
@@ -58,7 +59,7 @@ module ExcADG
       # @param payload Payload object to run in this Vertex
       # @param name optional vertex name to identify vertex
       # @param deps list of other Vertices or names to wait for
-      # @param timeout total time in seconds for the payload to run
+      # @param timeout {ExcADG::VTimeouts} or total time in seconds for the payload to run
       # @raise Payload::IncorrectPayloadArity in case payload returns function with arity > 1
       # @raise Payload::NoPayloadSet in case payload provided has incorrect type
       def new payload:, name: nil, deps: [], timeout: nil
@@ -66,33 +67,42 @@ module ExcADG
 
         raise Payload::IncorrectPayloadArity, "arity is #{payload.get.arity}, supported only 0 and 1" unless [0, 1].include? payload.get.arity
 
-        super(payload, name, timeout, DependencyManager.new(deps:)) { |payload, name, timeout, deps_manager|
+        dm = DependencyManager.new(deps: deps)
+        vtimeout = timeout.is_a?(VTimeout) ? timeout : VTimeout.new(payload: timeout)
+
+        super(payload, name, vtimeout, dm) { |payload, name, vtimeout, deps_manager|
           state_machine = StateMachine.new(name: name || "v#{number}".to_sym)
-          Broker.ask Request::Update.new data: state_machine.state_data
-          Log.info 'building vertex lifecycle'
-          state_machine.bind_action(:new, :ready) {
-            until deps_manager.deps.empty?
-              deps_data = Broker.ask Request::GetStateData.new(deps: deps_manager.deps)
-              deps_manager.deduct_deps deps_data
-              sleep 0.2
-            end
-            deps_manager.data
-          }
-          state_machine.bind_action(:ready, :done) {
-            function = payload.get
-            await(timeout:) {
-              case function.arity
-              when 0 then function.call
-              when 1 then function.call state_machine.state_data.data
-              else
-                raise Payload::IncorrectPayloadArity, "unexpected payload arity: #{function.arity}, supported only 0 and 1"
-              end
-            }
-          }
+          state_machine.with_fault_processing {
+            await(timeout: vtimeout.global) {
+              Broker.ask Request::Update.new data: state_machine.state_data
+              Log.info 'building vertex lifecycle'
+              state_machine.bind_action(:new, :ready) {
+                await(timeout: vtimeout.deps) {
+                  until deps_manager.deps.empty?
+                    deps_data = Broker.ask Request::GetStateData.new(deps: deps_manager.deps)
+                    deps_manager.deduct_deps deps_data
+                    sleep 0.2
+                  end
+                  deps_manager.data
+                }
+              }
+              state_machine.bind_action(:ready, :done) {
+                function = payload.get
+                await(timeout: vtimeout.payload) {
+                  case function.arity
+                  when 0 then function.call
+                  when 1 then function.call state_machine.state_data.data
+                  else
+                    raise Payload::IncorrectPayloadArity, "unexpected payload arity: #{function.arity}, supported only 0 and 1"
+                  end
+                }
+              }
 
-          Log.debug "another step fades: #{state_machine.state_data}" while state_machine.step
+              Log.debug "another step fades: #{state_machine.state_data}" while state_machine.step
 
-          Log.debug 'shut down'
+              Log.debug 'shut down'
+            }
+          }
         }
       end
     end

data/lib/excadg/vtimeout.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+module ExcADG
+  # complex vertex timeout to carry fine-grained timeouts
+  class VTimeout
+    attr_reader :global, :payload, :deps
+
+    # @param global timeout for the whole vertex in seconds
+    # @param deps for how long to wait for deps to complete in seconds
+    # @param payload how long to wait for payload in seconds
+    def initialize payload: nil, deps: nil, global: nil
+      if !global.nil? && global < (deps || 0) + (payload || 0) && (global < deps + payload)
+        raise "global timeout (#{global}) is less than sum of deps (#{deps}) and payload (#{payload}) timeouts"
+      end
+
+      @global = global
+      @deps = deps
+      @payload = payload
+    end
+  end
+end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: excadg
 version: !ruby/object:Gem::Version
-  version: 0.1.4
+  version: 0.2.0
 platform: ruby
 authors:
 - skorobogatydmitry
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-07-08 00:00:00.000000000 Z
+date: 2024-07-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rgl
@@ -51,6 +51,7 @@ files:
 - lib/excadg/tui.rb
 - lib/excadg/vertex.rb
 - lib/excadg/vstate_data.rb
+- lib/excadg/vtimeout.rb
 homepage: https://rubygems.org/gems/excadg
 licenses:
 - LGPL-3.0-only