excadg 0.2.5 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1c5c1e64e73685772165f04a188070729bedb2e4ef0ece0649815f9a3e56f182
-  data.tar.gz: f26fb805c50f800369b8402f1393c2dd7dc90b44093e2dda695ad956e9d895d8
+  metadata.gz: f27b6553ab8e3c1e7e77fadfeb9a4cdae08cd950063c7ab42436294c01fbae93
+  data.tar.gz: '06389a1abf75c660f102b6dd4675bd4af7f21014d674c724215dd13525ffed9e'
 SHA512:
-  metadata.gz: b60769a130cad0d93f40a4bebe239dc872046bda5f624747451781f78223d2551c992af71033abe8a7e19e40c0a8ce495703f206e9057bc82794bd4868e9e4f7
-  data.tar.gz: 4dfd3ad001c4e10907599a745fca844642fc20cdd138fddf5d829519f1c2b89e84c2fbd961d8b6807759aaf2f0f55de585ab2f9af1b9fdd0277c2db4005c7e4e
+  metadata.gz: 66431e9b380cfb4027dfa439566ada4e32e013e468a11deeced5be56f9da5668f9f9cedbb2d62a91eb570c61f805d276bbd98058cc867560fd75eebdf0f8b47b
+  data.tar.gz: c77bbe41f90c1d939d2a8b3c129abe1258aba0058e1a1e8ccfb4ce797b14dc4aa45c0803d91544c7504abfe23178fc824254a9146cb6b95dda04287f208e69d3

data/README.md

@@ -1,3 +1,28 @@
+- [Description](#description)
+- [Usage](#usage)
+  - [Tool](#tool)
+  - [Framework](#framework)
+    - [Payload implementation](#payload-implementation)
+    - [Vertices constructing](#vertices-constructing)
+  - [Tips](#tips)
+- [Internals](#internals)
+  - [Overview](#overview)
+  - [Vertice processing states](#vertice-processing-states)
+  - [{ExcADG::Broker}](#excadgbroker)
+  - [{ExcADG::DataStore}](#excadgdatastore)
+  - [{ExcADG::StateMachine}](#excadgstatemachine)
+  - [{ExcADG::Payload}](#excadgpayload)
+  - [{ExcADG::Log}](#excadglog)
+  - [{ExcADG::VTracker}](#excadgvtracker)
+- [Development](#development)
+  - [Gem](#gem)
+  - [Testing](#testing)
+- [Docs](#docs)
+- [Next](#next)
+  - [Core](#core)
+  - [TUI](#tui)
+  - [Payload](#payload)
+
 # Description
 
 That's a library (framework) to execute a graph of dependent tasks (vertices).  
@@ -8,11 +33,17 @@ Another feature is that the graph is dynamic and any vertex could produce anothe
 
 ## Tool
 
+tl;dr
+``` bash
+./bin/adgen --range 1:5 --file mygraph.yaml --count 30
+./bin/excadg --graph mygraph.yaml -l mygraph.log -d mygraph.yaml --gdump mygraph.jpg
+```
+
 There is a tool script in `bin` folder called `excadg`. Run `./bin/excadg --help` for available options.  
 It allows to run basic payload graphs specified by a YAML config. See [config/](config/) folder for sample configs.
 
 Another tool is `bin/adgen`, it has `--help` as well. It's suitable to generate relatively complex random graphs.  
-Try `./bin/adgen --range 1:5 --file mygraph.yaml --count 30` then `./bin/excadg --graph mygraph.yaml -l mygraph.log -d mygraph.yaml --gdump mygraph.jpg`.
+
 
 ## Framework
 
@@ -62,23 +93,27 @@ In this case, it'd be what `system 'echo here I am'` returns - `true`.
 
 Dependencies could be specified with both - {ExcADG::Vertex} objects and names. E.g.
 ``` ruby
-Broker.run
+Broker.instance.start
 
 v1 = Vertex.new payload: MyPayload.new
 Vertex.new name: :v2, payload: MyPayload.new
 
 Vertex.new name: :final, payload: MyPayload.new, deps: [v1, :v2]
 
-Broker.wait_all
+Broker.instance.wait_all
 ```
 
-*See [Broker section](#excadgbroker) for `Broker.run` and `Broker.wait_all` usage.*
+*See [Broker section](#excadgbroker) for `Broker.instance.start` and `Broker.instance.wait_all` usage.*
 
 Using actual objects looks simpler, but it's less convenient, as it requires you to construct all vertices as they appear in the execution graph.  
 However, names allows you to spawn vertices in arbitrary order and expect framework to figure execution order on the fly. See [run tool](#tool)'s code as an example of using names.
 
 *There is no need to store {ExcADG::Vertex} objects, as it and its data are available through [Broker](#excadgbroker)'s [DataStore](#excadgdatastore) and there is no interface to communicate with an {ExcADG::Vertex} directly.*
 
+## Tips
+
+If your app doesn't use all CPU cores with this library or has # of expected vertices much bigger than # of CPUs, try to `export RUBY_MAX_CPU=<num of cores> RUBY_MN_THREADS=1` to engage all cores. *See https://bugs.ruby-lang.org/issues/20618 for details.* 
+
 # Internals
 
 ## Overview
@@ -97,7 +132,7 @@ Internally, each vertice go through a sequence of states. Now it's **new**, **re
 {ExcADG::Vertex} starts as a **new** vertex and waits for its dependencies.  
 When vertex received all dependencies states and made sure they're **done**, it becomes **ready** and starts its payload.  
 When payload finishes, the vertex transitions to the **done** state.  
-If any of the stages fails, the vertex becomes **failed**. It could happen as due to a failed dependency (a stage failed) as well as any other error occurred (e.g. it receives an incorrect data from broker). A single failed vertex makes all vertices depending on it to fail. This way a failures cascading through the graph.
+If any of the stages fails, the vertex becomes **failed**. It could happen as due to a failed dependency (a stage failed) as well as any other error occurred (e.g. it receives an incorrect data from broker). A single failed vertex makes all vertices depending on it to fail. This way [failures cascading through the graph](config/faulty.yaml).
 
 ## {ExcADG::Broker}
 
@@ -107,10 +142,11 @@ When a vertex changes its state, it (actually, state machine does that) notifies
 
 Broker is desired to be as thin as possible to keep most of the work for vertices.
 
-Each application should invoke {Broker.run} to enable messages processing.  
-Its counterpart is {Broker.wait_all} which waits for all known vertices to reach one of the terminal states (**done** or **failed**) within specified timeout. Same as `Broker.run`, it spawns a thread and returns it. The main application could keep it in background and query or `.join` on it once all vertices are spawned. Once the thread finishes, the main app could lookup vertices execution results in {Broker.data_store} (see [DataStore](#excadgdatastore)).
+Each application should invoke `Broker.instance.start` to enable messages processing.  
+Its counterpart is `Broker.instance.wait_all` which waits for all known vertices to reach one of the terminal states (**done** or **failed**) within specified timeout. Same as `Broker.instance.start`, it spawns a thread and returns it. The main application could keep it in background and query or hang on `.join` once all vertices are spawned. Once the thread finishes, the main app could lookup vertices execution results in `Broker.instance.data_store` (see [DataStore](#excadgdatastore)). Broker could track all the seen vertices and their dependencies using builtin {ExcADG::VTracker}, add `track:true` to the `start` call to enable tracking.
 
-> Beware that broker constantly uses main Ractor's ports - incoming and outgoing. Hence, `Ractor#take`, `Ractor.yield` or any other messaging in the main ractor conflict with broker.
+> Note 1: tracking is a purely optional, broker itself requires {ExcADG::DataStore} only  
+> Note 2: beware that broker constantly uses main Ractor's ports - incoming and outgoing. Hence, `Ractor#take`, `Ractor.yield` or any other messaging in the main ractor conflict with broker.
 
 ## {ExcADG::DataStore}
 
@@ -141,6 +177,12 @@ Second way is built-in. Vertex invokes payload with {ExcADG::Array} of dependenc
 
 The library has its own logger based on Ractors. You could call {ExcADG::Log#unmute} to enable these logs.
 
+## {ExcADG::VTracker}
+
+It's an optional component which allows to track all vertices to be able to repro the full graph. There is no central place that stores the whole graph due to the ExcADG's core princilple - allow to spawn vertices at any time from any place. This class is introduced in order to support doing basic execution visualization (see {ExcADG::Tui}) and analysis.
+
+The tracker is integrated to the Broker and can be accessed by `ExcADG::Broker.instance.vtracker`. It's disabled by default to speed-up execution, but can be enabled on borker's startup by `Broker.instance.start track: true`.
+
 # Development
 
 The project is based on RVM => there is a .ruby-gemset file.  
@@ -180,14 +222,24 @@ Logging is disabled by default, but it could be useful to debug tests. Add `ExcA
 
 > here is a list of improvemets could be implementex next
 
-- implement throttling (:suspended state)
-  - limit # of running vertices
-    - problem: can't find what to suspend
-- make a loop payload template
-  - provide a mechanism to control # of children
-
-## Graph checks
-
-  - check for loops in the config
-  - check for unreachable islands - graph connectivity
-  - check that there are nodes to start from
+## Core
+1. implement throttling: allow to `:suspend` vertices that polls deps
+  - limit # of simultaneously running vertices
+  - limit # of vertices allowed to spawn
+2. implement checks using tracker
+  - for loops (it also leads to cases when there are no nodes to start from)
+  - for unreachable islands (optional, it could be expected)
+  - for the failure root cause
+3. move Vertice's tests to system test suite, make UTs for Vertice class
+
+## TUI
+1. make timeouts more flexible in the excadg tool
+2. allow to focus on a certain vertex to see what's it waiting for and what's waiting for it
+3. allow to dump focused vertice's state
+4. improve messages for timed out execution
+5. split TUI to another gem
+
+## Payload
+4. make a loop payload template
+5. allow to stream shell payload stdout/err in realtime to logs and files
+6. add native ruby example with a dynamic graph

data/bin/excadg

@@ -43,7 +43,7 @@ OptionParser.new { |opts|
   opts.on('-d', '--dump FILENAME', 'dump all vertices state data to the file in the end') { |o|
     options[:dump] = o
   }
-  opts.on('--gdump FILENAME', 'dump initial execution graph to the file specified') { |o|
+  opts.on('--gdump FILENAME', 'dump final execution graph to the file specified') { |o|
     options[:gdump] = o
   }
   opts.on('-t', '--timeout SECONDS', 'for how long to wait for the vertices to execute') { |o|
@@ -75,55 +75,56 @@ raise ArgumentError, 'at least one vertex should be ready to start' if runnable_
 
 ExcADG::Log.info 'collect execution graph from config'
 
-all_vertex_names = config.keys.collect!(&:to_sym)
-graph = RGL::DirectedAdjacencyGraph.new
 config.each_pair { |id, vconfig|
   name = id.to_sym
   payload = if vconfig&.key?('command')
               ExcADG::Payload::Wrapper::Bin.new args: vconfig['command']
             elsif vconfig&.key?('sleep')
               ExcADG::Payload::Example::Sleepy.new args: vconfig['sleep'].to_i
+            elsif vconfig&.key?('fail')
+              ExcADG::Payload::Example::Faulty.new args: vconfig['fail'] || 'injected failure'
             else
               ExcADG::Payload::Example::Echo.new args: vconfig
             end
   deps_v_names = (vconfig&.dig('dependencies') || []).collect(&:to_sym)
-  deps_v_names.each { |dep|
-    raise "there is no dependency named '#{dep}' in the graph for vertex '#{id}'" unless all_vertex_names.include? dep
-
-    graph.add_edge name, dep
-  }
-
-  ExcADG::Vertex.new name:, payload:, deps: deps_v_names
+  ExcADG::Vertex.new name:, payload:, deps: deps_v_names, timeout: options[:timeout]
 }
 
-graph.write_to_graphic_file(options[:gdump].split('.').last, options[:gdump].split('.')[...-1].join('.')) unless options[:gdump].nil?
-
 ExcADG::Log.info 'starting state data broker'
-ExcADG::Broker.run
+ExcADG::Broker.instance.start track: true
 
 ui_drawer&.run
 
 ExcADG::Log.info 'watching for all vertices to complete'
 timed_out = false
 begin
-  ExcADG::Broker.wait_all(timeout: options[:timeout]).join
+  waiter = ExcADG::Broker.instance.wait_all(timeout: options[:timeout]).join
 rescue Timeout::Error
   ExcADG::Log.error 'execution timed out'
   timed_out = true
+rescue Interrupt
+  ExcADG::Broker.instance.teardown
+  waiter&.kill
 end
-vertice_by_state = ExcADG::Broker.data_store.to_a.group_by(&:state)
 
-ExcADG::Log.info "#{vertice_by_state[:done]&.size || 0} done, #{vertice_by_state[:failed]&.size || 0} failed"
-has_failed = !vertice_by_state[:failed].nil?
+ExcADG::Log.info "vertice counts by state: #{ExcADG::Broker.instance.vtracker.by_state.transform_values(&:size)}"
+has_failed = ExcADG::Broker.instance.vtracker.by_state.key? :failed
 ui_drawer&.summarize has_failed, timed_out
 
 unless options[:dump].nil?
   ExcADG::Log.info "writing data to #{options[:dump]}"
   File.open(options[:dump], 'w+') { |f|
-    f.write JSON.dump ExcADG::Broker.data_store.to_a
+    f.write JSON.dump ExcADG::Broker.instance.data_store.to_a
   }
 end
 
+unless options[:gdump].nil?
+  extension = options[:gdump].split('.').last
+  name = options[:gdump].split('.')[...-1].join('.')
+  ExcADG::Broker.instance.vtracker.graph.write_to_graphic_file(extension, name)
+  ExcADG::Log.debug "graph is saved to #{options[:gdump]}"
+end
+
 sleep 0.1 # let the logger to print message
 exit_code = 0
 exit_code |= 1 if has_failed

data/lib/excadg/broker.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require 'timeout'
+require 'singleton'
 
 require_relative 'data_store'
 require_relative 'log'
@@ -9,91 +10,121 @@ require_relative 'vertex'
 
 module ExcADG
   # handle requests sending/receiving though Ractor's interface
-  module Broker
-    class << self
-      attr_reader :data_store
-
-      # is used from vertices to send reaqests to the broker
-      # @return data received in response from the main ractor
-      # @raise StandardError if response is a StandardError
-      # @raise CantSendRequest if sending request failed for any reason
-      def ask request
-        raise UnknownRequestType, request unless request.is_a? Request
-
-        begin
-          Ractor.main.send request
-        rescue StandardError => e
-          raise CantSendRequest, cause: e
-        end
-
-        Log.info 'getting response'
-        resp = Ractor.receive
-        Log.debug "got response #{resp}"
-        raise resp if resp.is_a? StandardError
-
-        Log.debug 'returning response'
-        resp
+  class Broker
+    include Singleton
+
+    attr_reader :data_store, :vtracker
+
+    # is used from vertices to send requests to the broker
+    # @param request {Request}
+    # @return data received in response from the main ractor
+    # @raise {StandardError} if response is a StandardError
+    # @raise {CantSendRequest} if sending request failed for any reason
+    # @raise {UnknownRequestType} if request is not a {Request}
+    def self.ask request
+      raise UnknownRequestType, request unless request.is_a? Request
+
+      begin
+        Ractor.main.send request
+      rescue StandardError => e
+        raise CantSendRequest, cause: e
       end
 
-      # start requests broker for vertices in a separated thread
-      # @return the thread started
-      def run
-        @data_store ||= DataStore.new
-        @broker = Thread.new { loop { process_request } } unless @broker&.alive?
+      Log.info 'getting response'
+      resp = Ractor.receive
+      Log.debug "got response #{resp}"
+      raise resp if resp.is_a? StandardError
 
-        at_exit {
-          Log.info 'shut down broker'
-          @broker.kill
-        }
+      Log.debug 'returning response'
+      resp
+    end
 
-        Log.info 'broker is started'
-        @broker
-      end
+    # start processing vertices asks
+    # @param track whether to track vertices using {VTracker}
+    # @return messanges processing {Thread}
+    def start track: true
+      @vtracker = VTracker.new if track
+      Thread.report_on_exception = false
+      @messenger = Thread.new { loop { process_request } }
+
+      at_exit {
+        Log.info 'shutting down messenger'
+        @messenger.kill
+        Log.info 'messenger is stut down'
+      }
+
+      Log.info 'broker is started'
+      @messenger
+    end
+
+    # stop processing vertices asks
+    # usually follows {#wait_all}
+    def teardown
+      return if @messenger.nil?
+
+      @messenger.kill while @messenger.alive?
+    end
 
-      # makes a thread to wait for all known vertices to reach a final state
-      # @param timeout total waiting timeout in seconds, nil means wait forever
-      # @param period time between vertices state check
-      def wait_all timeout: 60, period: 1
-        Thread.new {
-          Log.info "timeout is #{timeout || '∞'} seconds"
-          Timeout.timeout(timeout) {
-            loop {
-              sleep period
-              states = @data_store.to_a.group_by(&:state).keys
-              Log.info "vertices in #{states} exist"
-              # that's the only final states for vertices
-              break if (states - %i[done failed]).empty?
-            }
+    # makes a thread to wait for all known vertices to reach a final state;
+    # it expects some vertices to be started in the outer scope,
+    # so it waits even if there are no vertices at all yet
+    # @param timeout total waiting timeout in seconds, nil means wait forever
+    # @param period time between vertices state check
+    # @return {Thread} that waits for all deps, typical usage is `Broker.instance.wait_all.join`
+    def wait_all timeout: 60, period: 1
+      Thread.report_on_exception = false
+      Thread.new {
+        Log.info "timeout is #{timeout || '∞'} seconds"
+        Timeout.timeout(timeout) {
+          loop {
+            sleep period
+            if @data_store.empty?
+              Log.info 'no vertices in data store, keep waiting'
+              next
+            end
+            states = @data_store.to_a.group_by(&:state).keys
+            Log.info "vertices in #{states} states exist"
+            # that's the only final states for vertices
+            break if (states - %i[done failed]).empty?
           }
         }
-      end
+      }
+    end
 
-      private
-
-      # waits for an incoming request,
-      # validates request type and content
-      # then makes and sends an answer
-      def process_request
-        begin
-          request = Ractor.receive
-          Log.info "received request: #{request}"
-          request.self.send case request
-                            when Request::GetStateData
-                              request.filter? ? request.deps.collect { |d| @data_store[d] } : @data_store.to_a
-                            when Request::Update
-                              @data_store << request.data
-                              true
-                            when Request::AddVertex
-                              Vertex.new payload: request.payload, deps: [request.self]
-                            else
-                              raise UnknownRequestType
-                            end
-        rescue StandardError => e
-          Log.warn "error on message processing: #{e.class} / #{e.message} / #{e.backtrace}"
-          request.self.send RequestProcessingFailed.new cause: e
-        end
-        Log.debug 'message processed'
-      end
+    private
+
+    def initialize
+      @data_store = DataStore.new
+    end
+
+    # waits for an incoming request,
+    # validates request type and content
+    # then constructs and sends an answer
+    # does not fail on {StandardError}, logs & sends it back instead to keep running;
+    # the other side crashes in this case, causing the {Vertex} to fail, what's still
+    # better than crashing the whole messaging
+    def process_request
+      request = Ractor.receive
+      Log.info "received request: #{request}"
+      request.self.send case request
+                        when Request::GetStateData
+                          @vtracker&.track request.self, request.deps
+                          request.filter? ? request.deps.collect { |d| @data_store[d] } : @data_store.to_a
+                        when Request::Update
+                          @data_store << request.data
+                          @vtracker&.track request.self
+                          true
+                        when Request::AddVertex
+                          v = Vertex.new payload: request.payload, deps: [request.self]
+                          @vtracker&.track v
+                          v
+                        else
+                          raise UnknownRequestType
+                        end
+    rescue StandardError => e
+      # TODO: add threshold
+      Log.warn "error on message processing: #{e.class} / #{e.message} / #{e.backtrace}"
+      request.self.send RequestProcessingFailed.new cause: e
     end
 
     class UnknownRequestType < StandardError; end
@@ -101,11 +132,12 @@ module ExcADG
 
     # error type returned by broker thread in case it failed to process incoming request
     class RequestProcessingFailed < StandardError
-      attr_reader :cause
+      attr_reader :cause_backtrace
 
       def initialize cause:
-        super
-        @cause = cause
+        # storing the cause itself could creak messaging, as it could contain non-shareable objects
+        super cause.message
+        set_backtrace cause.backtrace
       end
     end
   end

data/lib/excadg/data_store.rb

@@ -6,8 +6,6 @@ require_relative 'vstate_data'
 module ExcADG
   # collection of {ExcADG::VStateData} for {ExcADG::Broker}
   class DataStore
-    attr_reader :size
-
     def initialize
       # two hashes to store VStateData and access them fast by either key
       @by_name = {}
@@ -75,6 +73,10 @@ module ExcADG
       end
     end
 
+    def empty?
+      @size.zero?
+    end
+
     class DataSkew < StandardError; end
   end
 end

data/lib/excadg/log.rb

@@ -29,7 +29,7 @@ module ExcADG
     end
 
     # default logger
-    @main = RLogger.new
+    @main = nil
 
     # logging is muted by default
     @muted = true
@@ -37,11 +37,12 @@ module ExcADG
     def self.method_missing(method, *args, &_block)
       return if @muted
 
+      @main ||= RLogger.new
       r = Ractor.current
       @main.send [method, r&.to_s || r.object_id, *args]
     rescue Ractor::ClosedError => e
       # last hope - there is tty
-      puts "can't send message to logging ractor: #{e}"
+      puts "can't send message to logging ractor: #{e}, message: #{args}"
     end
 
     def self.respond_to_missing?

data/lib/excadg/payload/wrapper.rb

@@ -13,21 +13,20 @@ module ExcADG
       include Payload
       def get
         lambda { |deps_data|
-          begin
-            f = nil
             Dir.mktmpdir { |dir|
-              f = File.new File.join(dir, 'data.json'), 'w+'
-              f.write JSON.generate deps_data
-              f.flush
-              stdout, stderr, status = Open3.capture3({ 'DEPS_DATAFILE' => f.path }, args)
+              Log.debug "temp dir #{} for data is ready"
+              stdout, stderr, status = File.open(File.join(dir, 'data.json'), 'w+') { |f|
+                f.write JSON.generate deps_data
+                f.flush
+                Log.debug "data is in #{f.path}"
+                Open3.capture3({ 'DEPS_DATAFILE' => f.path }, args)
+              }
+              Log.debug "payload process finished"
               data = { stdout:, stderr:, exitcode: status.exitstatus }
               raise CommandFailed, data unless status.exitstatus.zero?
-
+              Log.debug "returning data"
               data
             }
-          ensure
-            f&.close
-          end
         }
       end
 

data/lib/excadg/request.rb

@@ -13,11 +13,15 @@ module ExcADG
       @self = Ractor.current
     end
 
+    def to_s
+      "#{self.class} from #{@self}"
+    end
+
     # request to get state data
     class GetStateData < ExcADG::Request
       attr_reader :deps
 
-      # @param deps Array of VStateData::Key
+      # @param deps {Array} of VStateData::Key
       def initialize deps: nil
         super()
         @deps = deps

data/lib/excadg/rtimeout.rb

@@ -11,13 +11,14 @@ module ExcADG
       timed_out = false
       Thread.report_on_exception = false
       payload = Thread.new { Thread.current[:result] = block.call }
-      Thread.new {
+      countdown_t = Thread.new {
         sleep timeout
         payload.kill
         timed_out = true
       }
 
       payload.join
+      countdown_t.kill
       timed_out ? raise(TimedOutError) : payload[:result]
     end
     module_function :await

data/lib/excadg/state_machine.rb

@@ -55,6 +55,7 @@ module ExcADG
       edge = GRAPH.edges.find { |e| e.source == @state && e.target = target }
 
       with_fault_processing {
+        Log.debug "calling payload for #{edge}"
         @state_transition_data[target] = @state_edge_bindings[edge].call
         @state = target
         Log.debug "moved to #{@state}"
@@ -66,7 +67,7 @@ module ExcADG
     def with_fault_processing
       yield if block_given?
     rescue StandardError => e
-      Log.error "step failed with #{e} / #{e.backtrace}"
+      Log.error "step failed with '#{e}' / #{e.backtrace}"
       @state_transition_data[:failed] = e
       @state = :failed
     ensure

data/lib/excadg/tui/block.rb

@@ -116,11 +116,11 @@ module ExcADG::Tui
     # {#column} and {#row} are enough to make blocks;
     # in case you need to align a single block, use e.g. `Block.column("one", "two") { |blk| blk.box!.pad! 2 }`
     def initialize arg
-      case arg
-      when Array then @array = arg
-      when String then @array = [arg]
-      else raise "can't make block from #{arg.class}"
-      end
+      @array = case arg
+               when Array then arg
+               when String then [arg]
+               else [arg.to_s]
+               end
       @width = @array.collect(&:size).max
     end
   end

data/lib/excadg/tui/format.rb

@@ -9,7 +9,7 @@ module ExcADG::Tui
   module Format
     # add horizontal padding to the block
     # @param size number of spaces to add
-    def h_pad! size
+    def h_pad! size = 1
       @array.collect! { |row|
         "#{' ' * size}#{row}#{' ' * size}"
       }
@@ -19,7 +19,7 @@ module ExcADG::Tui
 
     # add vertical padding to the block
     # @param size number of spaces to add
-    def v_pad! size
+    def v_pad! size = 1
       filler = ' ' * @width
       size.times {
         self >> filler
@@ -30,7 +30,7 @@ module ExcADG::Tui
 
     # adds spaces around the block
     # @param size number of spaces to add
-    def pad! size
+    def pad! size = 1
       v_pad! size
       h_pad! size
     end

data/lib/excadg/tui.rb

@@ -13,7 +13,7 @@ module ExcADG
   module Tui
     MAX_VERTEX_TO_SHOW = 10
     DELAY = 0.2
-    DEFAULT_BOX_SIZE = { height: 50, width: 150 }.freeze
+    DEFAULT_BOX_SIZE = { height: 70, width: 200 }.freeze
 
     @started_at = DateTime.now.strftime('%Q').to_i
 
@@ -21,6 +21,7 @@ module ExcADG
       # spawns a thread to show stats to console in background
       def run
         Log.info 'spawning tui'
+        Thread.report_on_exception = false
         @thread = Thread.new {
           loop {
             # print_in_box stats
@@ -41,22 +42,26 @@ module ExcADG
       # private
 
       def get_summary has_failed, timed_out
-        [timed_out ? 'execution timed out' : 'execution completed',
-         "#{has_failed ? 'some' : 'no'} vertices failed"]
+        [timed_out ? 'execution timed out' : '🮱  execution completed',
+         "#{has_failed ? '🯀  some' : '🮱  no'} vertices failed"]
       end
 
       # make summary paragraph on veritces
       def stats summary: nil
         Block.column(
-          Block.column(summary || 'running').h_pad!(1).box!.v_align!(:center, width: @content_size[:width]),
-          Block.column(
-            *[
-              "time spent (ms): #{DateTime.now.strftime('%Q').to_i - @started_at}",
-              "vertices seen: #{Broker.data_store.size}",
-              'progress:'
-            ] + state_stats,
-            align: :left
-          ).h_pad!(2),
+          Block.row(
+            Block.column(summary || '🮚  running').h_pad!(1).v_align!(:center).box!.h_pad!,
+            Block.column(
+              *[
+                "🮚  time spent (ms): #{DateTime.now.strftime('%Q').to_i - @started_at}",
+                "#  vertices seen: #{Broker.instance.vtracker.graph.vertices.size}",
+                '🮶  progress:'
+              ] + state_stats,
+              align: :left
+            ).h_pad!(2)
+          ),
+          Block.row('🮲 🮳  pending vertices and their dependencies:').pad!,
+          Block.row(*pending_vertices || 'tracking is n/a', align: :top).h_pad!,
           align: :left
         ).fit!(width: @content_size[:width], height: @content_size[:height], fill: true)
              .box!(corners: :sharp)
@@ -67,31 +72,38 @@ module ExcADG
       end
 
       def refresh_sizes
-        box_size = {
-          height: IO.console&.winsize&.first.nil? || DEFAULT_BOX_SIZE[:height] < IO.console.winsize.first ? DEFAULT_BOX_SIZE[:height] : IO.console.winsize.first,
-          width: IO.console&.winsize&.last.nil? || DEFAULT_BOX_SIZE[:width] < IO.console&.winsize&.last ? DEFAULT_BOX_SIZE[:width] : IO.console.winsize.last
-        }.freeze
         @content_size = {
-          height: box_size[:height] - 4, # 2 for borders, 1 for \n, 1 for remark
-          width: box_size[:width] - 5 # 2 for borders, 2 to indent
-        }.freeze
-        @line_template = "│ %-#{@content_size[:width]}s │\n"
+          height: (IO.console&.winsize&.first.nil? || DEFAULT_BOX_SIZE[:height] < IO.console.winsize.first ? DEFAULT_BOX_SIZE[:height] : IO.console.winsize.first) - 4,
+          width: (IO.console&.winsize&.last.nil? || DEFAULT_BOX_SIZE[:width] < IO.console&.winsize&.last ? DEFAULT_BOX_SIZE[:width] : IO.console.winsize.last) - 2
+        }
       end
 
       # make states summary, one for a line with consistent placing
       def state_stats
         skeleton = StateMachine::GRAPH.vertices.collect { |v| [v, []] }.to_h
-        # rubocop:disable Style/HashTransformValues
-        filled = skeleton.merge Broker.data_store.to_a
-                                      .group_by(&:state)
-                                      .collect { |state, vertices| [state, vertices_stats(vertices)] }
-                                      .to_h
-        # rubocop:enable Style/HashTransformValues
+        filled = skeleton.merge(Broker.instance.vtracker.by_state.transform_values { |vertices| vertices.collect(&:name).join(', ') })
         filled.collect { |k, v| format '  %-10s: %s', k, "#{v.empty? ? '<none>' : v}" }
       end
 
-      def vertices_stats vertice_pairs
-        vertice_pairs.collect(&:name).join(', ')
+      # gather pending vertices (in :new state) from the tracker
+      # and render dependencies they're waiting for
+      def pending_vertices
+        return nil if Broker.instance.vtracker.nil?
+        if Broker.instance.vtracker.by_state[:new].nil? || Broker.instance.vtracker.by_state[:new].empty?
+          return Block.column('... no pending vertices').h_pad!
+        end
+
+        Broker.instance.vtracker.by_state[:new].sort_by(&:name).collect { |v|
+          deps = Broker.instance.vtracker.get_deps v
+          deps = nil if deps.empty?
+          deps&.sort_by!(&:state)
+          width = 20
+          Block.column(
+            Block.column(v).fit!(width: width - 4).h_pad!.box!,
+            '🮦',
+            Block.column(*deps || 'no deps to wait').fit!(width: width - 4).h_pad!.box!
+          ).fit!(width: width, fill: true).h_pad!(2)
+        }
       end
     end
   end

data/lib/excadg/vertex.rb

@@ -30,8 +30,14 @@ module ExcADG
       info&.dig(0).to_i || -1
     end
 
+    # only main ractor could try to get real symbolic vertex name and state
+    # other ractors (e.g. logger) could only use builtin data
     def to_s
-      "#{number} #{status}"
+      if Ractor.current == Ractor.main
+        "#{name || number} #{state || status}"
+      else
+        "#{number} #{status}"
+      end
     end
 
     # below are shortcut methods to access Vertex data from the main Ractor
@@ -39,7 +45,7 @@ module ExcADG
     # obtains current Vertex-es data by lookup in the Broker's data,
     # available from the main Ractor only
     def data
-      Broker.data_store[self]
+      Broker.instance.data_store&.[](self)
     end
 
     # gets current Vertex's state,
@@ -88,13 +94,16 @@ module ExcADG
               }
               state_machine.bind_action(:ready, :done) {
                 function = payload.get
+                Log.debug "payload has arity #{function.arity}"
                 await(timeout: vtimeout.payload) {
-                  case function.arity
-                  when 0 then function.call
-                  when 1 then function.call state_machine.state_data.data
-                  else
-                    raise Payload::IncorrectPayloadArity, "unexpected payload arity: #{function.arity}, supported only 0 and 1"
-                  end
+                  result = case function.arity
+                           when 0 then function.call
+                           when 1 then function.call state_machine.state_data.data
+                           else
+                             raise Payload::IncorrectPayloadArity, "unexpected payload arity: #{function.arity}, supported only 0 and 1"
+                           end
+                  Log.debug 'payload finished'
+                  result
                 }
               }
 

data/lib/excadg/vtracker.rb

@@ -4,7 +4,7 @@ require 'rgl/adjacency'
 
 module ExcADG
   # tracker for {Vertex}-es graph:
-  # it's hooked by {Broker} to register dependencies polling events
+  # it's hooked by {Broker} to register dependencies polling (and other) events
   # and make an actual graph of vertices with states in runtime
   #
   # it's not possible to do this in other way, because vertices can be spawned:
@@ -21,22 +21,29 @@ module ExcADG
     # register the vertex and its new known deps in the @graph and by_state cache
     # @param vertice vertice that requested info about deps
     # @param deps list of dependencies as supplied by {Request::GetStateData}
-    def track vertex, deps
+    def track vertex, deps = []
       Assertions.is_a? vertex, Vertex
       Assertions.is_a? deps, Array
 
+      @graph.add_vertex vertex
       add_to_states_cache vertex, vertex.state
 
       deps.each { |raw_dep|
-        # it could be not Vertex, so do a lookup through data store
-        next unless Broker.data_store[raw_dep]
+        # it could be not a Vertex, so do a lookup through data store
+        next unless Broker.instance.data_store[raw_dep]
 
-        dep_data = Broker.data_store[raw_dep]
+        dep_data = Broker.instance.data_store[raw_dep]
         add_to_states_cache dep_data.vertex, dep_data.state
         @graph.add_edge vertex, dep_data.vertex
       }
     end
 
+    # get all vertex's dependencies
+    # @param vertex {Vertex} vertex to lookup known dependencies for
+    def get_deps vertex
+      @graph.adjacent_vertices vertex
+    end
+
     private
 
     # adds a given {Vertex} in state to @by_state cache;

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: excadg
 version: !ruby/object:Gem::Version
-  version: 0.2.5
+  version: 0.4.0
 platform: ruby
 authors:
 - skorobogatydmitry
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-07-31 00:00:00.000000000 Z
+date: 2024-08-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rgl