excadg 0.1.0 → 0.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e1d50ffd84523c400ae4733e67c53d54a791cb066cec4a49dea27b4e7072ecda
-  data.tar.gz: 4b5bf03ce6c2e9190e5af55a9f740247281da6ed6522c37a7ee3dd3698352dc9
+  metadata.gz: 6eaa4affddc5100a387f6faa4006451eac4fe7ca15d03817fd962e25b25c8a79
+  data.tar.gz: 607ad2df4d09b404ab728d3397f6a82b41631a9a603412367f66dabbba1e2a1f
 SHA512:
-  metadata.gz: 274e5395a4d9ed5da1c5aba9a368638757f1571ac38aede22b0f729a794813c4611e41dfb901a9d555c0a202d7b494b47e077b9c379383a7cb116c2f1dd32ecd
-  data.tar.gz: e35cd2c39d26791742b1c9ff1255f3c2edb9247b046a09dad62013c6bd9e48fdae54be73e17ae4e1fb430a385a2ec8bc19cfba63a54deffb21b27d59a196da50
+  metadata.gz: 9b540b1c30fe6a6a1a124e55a29dd21d68d2cf27631a6bfce4e14fa41fc2708e57587eb1174ba1ec15ce75be66eb7d8b54da392dedfc6afca63fd35fd56bf3ce
+  data.tar.gz: c16ae1cf5e5c723cad55a44ff2e16881ab575145cabb2a6ff2d75ba5ab935249cfa79fd22ab4b7a1966adc02cbf03674a57fe1453b8add297b16a7be6ed93303

data/README.md

@@ -0,0 +1,189 @@
+# Description
+
+That's a library (framework) to execute a graph of dependent tasks (vertices).  
+Its main feature is to run all possible tasks independently in parallel once they're ready.  
+Another feature is that the graph is dynamic and any vertex could produce another vertice(s) to extend execution graph.
+
+# Usage
+
+## Tool
+
+There is a tool script in root folder called `excadg`. Run `./bin/excadg --help` for available options.  
+It allows to make and run basic payload graphs specified by a YAML config. See [config/](config/) folder for sample configs.
+
+## Framework
+
+Main usage pattern for this project is a framework. It implies that you:
+- [implement custom payload](#payload-implementation) for vertices
+- write a script or extend your program to [construct a swarm vertices](#vertices-constructing) to execute
+
+
+### Payload implementation
+
+Payload is a piece of code to execute in a single vertex. This code should be self-contained and use specific ways to communicate with other vertices.  
+There is a [{ExcADG::Payload}](lib/excadg/payload.rb) module to make a custom payload class. Read through its documentation to get familiar with its interface.  
+
+Here is a very basic example of a payload that echoes a message:
+
+```
+class MyPayload
+  include ExcADG::Payload
+  def get
+   lambda { system 'echo here I am' }
+  end
+end
+```
+More payload examples could be found in [{ExcADG::Payload::Example}](lib/excadg/payload/example.rb).
+
+Any payload has 3 parts:
+- code
+- arguments
+- dependencies data
+
+**Code** is defined in {Payload#get} method. E.g. `system 'echo here I am'` in the above example.  
+**Arguments** are defined during payload object contruction (see {Payload#initialize}) as `args:` parameter. E.g. `MyPayload.new args: 'my custom message'`.  
+**Dependencies data** is provided by the framework and has all [{ExcADG::VStateData}](lib/excadg/vstate_data.rb) returned by the dependencies.
+
+### Vertices constructing
+
+Vertex is a execution graph's building block. It executes its payload when all its dependencies are finished.  
+Vertex can be created this way: `Vertex.new name: :myvertex, payload: MyPayload.new`. This vertex doesn't have any dependencies, so it starts execution immediately upon construction. Vertices without dependencies are always aneeded in the graph to start it.  
+
+Another "kind" of vertices are vertices with dependencies. Dependencies are other vertices that the vertex waits to finish successfully before running its own payload.  
+> Example: `Vertex.new name: :other_vertex, payload: MyPayload.new, deps: [:myvertex]` has exactly 1 dependency - vertex called `:myvertex`  
+> `:other_vertex` won't start until `:myvertex` finishes.  
+> Moreover, as described in [payload](#payload), `:other_vertex`'s payload will have access to data returned by `:myvertex`'s payload.  
+> In this case, it'd be what `system 'echo here I am'` returns - `true`.
+
+Dependencies could be specified both - using {ExcADG::Vertex} objects and names. E.g.
+``` ruby
+Broker.run
+
+v1 = Vertex.new payload: MyPayload.new
+Vertex.new name: :v2, payload: MyPayload.new
+
+Vertex.new name: :final, payload: MyPayload.new, deps: [v1, :v2]
+
+Broker.wait_all
+```
+
+*See [Broker section](#broker) for `Broker.run` and `Broker.wait_all` usage.*
+
+Using actual objects looks simpler, but it's less convenient, as it requires you to construct all vertices as they appear in the execution graph.  
+However, names allows you to spawn vertices in arbitrary order and expect framework to figure execution order on the fly. See [run tool](#tool) as an example of using names.
+
+*There is no need to store {ExcADG::Vertex} objects, as it and its data are available through [Broker](#broker)'s [DataStore](#data-store) and there is no interface to communicate with a {ExcADG::Vertex} directly.*
+
+# Internals
+
+## Overview
+
+This framework allows to spawn vertices. Once created, a vertex with payload starts immediately.
+This means that there is no central place where execution seqence is controlled besides vertex's own mechanism to wait for dependencies.
+
+As Ractors doesn't allow to reliably exchange data between each other at the moment, the main Ractor (thread) has to spawn a broker to synchronize data exchange. See [broker](#broker) sections to learn more.
+
+This framework is based on [Ractors](https://docs.ruby-lang.org/en/master/ractor_md.html). It could be useful to get familiar with ractors before reading further.
+
+## Vertice processing states
+
+Internally, each vertice goes through a sequence of states. Now it's **new**, **ready**, **done** and **failed**. Stages are controlled by the [{ExcADG::StateMachine}](#statemachine).
+
+{ExcADG::Vertex} starts as a **new** vertex and waits for its dependencies.  
+When vertex received all dependencies states and made sure they're **done**, it becomes **ready** and starts its payload.  
+When payload finishes, the vertex transitions to the **done** state.  
+If any of the stages fails, the vertex becomes **failed**. It could happen as due to a failed dependency (a stage failed) as well as any other error occurred (e.g. it receives an incorrect data from broker). A single failed vertex makes all vertices depending on it to fail. This way a failures cascading through the graph.
+
+## {ExcADG::Broker}
+
+Broker is a central component meant to receive and transmit data between vertices. There are several [{ExcADG::Request}](lib/excadg/request.rb) types broker supports.
+
+When a vertex changes its state, it (actually, state machine does that) notifies the broker of its state and sends data (results of the transition) ptp the broker. Broker stores this data in a map and could send it to other vertices by request. Vertices polls their dependencies through broker to know where all dependencies are done or some of them failed.
+
+Broker is desired to be as thin as possible to keep most of the work for vertices.
+
+Each application should invoke {Broker.run} to enable messages processing.  
+Its counterpart is {Broker.wait_all} which waits for all known vertices to reach one of the terminal states (**done** or **failed**) within specified timeout. Same as `Broker.run`, it spawns a thread and returns it. The main application could keep it in background and query or `.join` on it once all vertices are spawned. Once the thread finishes, the main app could lookup vertices execution results in {Broker.data_store} (see [DataStore](#data-store)).
+
+> Beware that broker constantly uses main Ractor's ports - incoming and outgoing. Hence, `Ractor#take`, `Ractor.yield` or any other messaging in the main ractor conflict with broker.
+
+## {ExcADG::DataStore}
+
+It's a mostly internal [Broker's](#broker) object that holds all vertice's [{ExcADG::VStateData}](lib/excadg/vstate_data.rb).
+
+## {ExcADG::StateMachine}
+
+State machine is a small internal mechanism that helps vertices to
+- do state transitions
+- preserve state transition results locally
+
+State machine has transitions graph that is common for all vertices. Vertices bind certain logic to transitions. E.g. "wait for dependencies" or "run the payload". State transition mechanism ensures to run workload associated to the currently possible transition, process errors (if any) and send results to [Broker](#broker).
+
+## {ExcADG::Payload}
+
+Payload is a special module that carries convention for Vertex-es payload. Only classes that `include ExcADG::Payload` are expected to be providede as payload during vertex creation. _Although, obviously, there are dozens of ways to trick the code._
+
+Payload existence is caused by that Ractors require all objects a Ractor uses to be moved (or copied) to it. A pure {ExcADG::Proc} can't be transferred, as it doesn't have allocator and can access outer scope (which has to be transferred too then), what could be impossible due to other non-shareable objects there.
+
+To make the interface clearer and more reliable, Payload encloses a `Proc` within its `get` method. As isolated `Proc`s are not useful most of the time, Payload has 2 ways to provide/access the needful for payload to work.  
+First way is `args` parameter of the {Payload#initialize}. It can be used on payload creation time to customize payload's behavior. The paremeter is accessible within the labmda through `@args` attribute.  
+Second way is built-in. Vertex invokes payload with {ExcADG::Array} of dependencies {ExcADG::VStateData}, if payload's `Proc` is parametrized (has arity 1).
+
+> Be mindful about data your payload receives (args) and returns (state data). It could appear incompatible with ractors and cause vertice to fail.  
+> Although these failures are hard to tell beforehand, [state machine](#statemachine) processes them gracefully.
+
+# Development
+
+The project is based on RVM => there is a .ruby-gemset file.  
+Bundler is configured to install gems for dev environment, use `bundle install --without dev` to install only modules needed to run the code.
+
+## Gem
+
+There is a gem specification. Typical commands:
+- build gem: `gem build excadg.gemspec`
+- install / uninstall gem built locally: `gem install ./excadg*.gem` / `gem uninstall excadg`
+- publish: `gem push excadg*.gem`
+
+## Testing
+
+Test are located in [spec/](spec/) folder and are written using rspec.
+
+Commands to run tests:
+- `rspec` - run most of the tests
+- `rspec spec/broker_spec.rb` - run tests from the file
+- `rspec spec/broker_spec:32` - run tests that starts on line 32 of the file
+- `rspec --tag perf` - run tests of a specific suite
+
+  > there is no consistent set of suites, tags are used to exclude mutually incompatible tests or e.g. perfomance tests  
+  > search for `config.filter_run_excluding` in [spec/spec_helper.rb](spec/spec_helper.rb) to see what tests are disabled by default
+
+### Logging
+
+Most of the tests have loggin stubbed to avoid noize. Comment out either `stub_loogging` or `Log.mute` in the respective spec file to enable logging for it.
+
+# Docs
+
+`yard doc lib/` generates a descent documentation.
+
+`yard server --reload` starts a server with documentation available at http://localhost:8808
+
+# Next
+
+> here is a list of improvemets could be implementex next
+
+- add timeouts
+  - to payload
+  - to whole vertex
+  - to request processing
+- implement throttling (:suspended state)
+  - limit # of running vertices
+    - problem: can't find what to suspend
+- make a loop payload template
+  - provide a mechanism to control # of children
+- avoid initializing Log on require
+
+## Graph checks
+
+  - check for loops in the config
+  - check for unreachable islands - graph connectivity
+  - check that there are nodes to start from

data/bin/excadg

@@ -0,0 +1,133 @@
+#!/usr/bin/env ruby
+
+require 'yaml'
+require 'optparse'
+
+require 'rgl/adjacency'
+require 'rgl/dot'
+
+require_relative '../lib/excadg'
+
+options = {
+  graph: 'config.yaml',
+  logfile: :stdout,
+  loglevel: Logger::INFO,
+  dump: nil,
+  gdump: nil,
+  timeout: nil
+}
+OptionParser.new { |opts|
+  opts.banner = <<~EOD
+    tool to run a graph of payloads specified in config
+    usage: #{$PROGRAM_NAME} [args]
+  EOD
+  opts.on('-g', '--graph FILENAME', 'config file name to load, see config/sample.yaml for format', "default: #{options[:graph]}") { |o|
+    options[:graph] = o
+  }
+  opts.on(
+    '-l', '--outmode <FILENAME|:silent|:stdout>',
+    'output mode',
+    'app draws TUI if log file is set',
+    'reserved keywords for stdout and silent modes',
+    "default: #{options[:logfile]}"
+  ) { |o|
+    options[:logfile] = o
+  }
+  opts.on('--loglevel <DEBUG|INFO|WARN|ERROR|FATAL>',
+          "log level, default: #{options[:loglevel]}") { |o|
+    name = o.upcase.to_sym
+    raise "unknown log level #{o}" unless Logger.const_defined? name
+
+    options[:loglevel] = Logger.const_get name
+  }
+  opts.on('-d', '--dump FILENAME', 'dump all vertices state data to the file in the end') { |o|
+    options[:dump] = o
+  }
+  opts.on('--gdump FILENAME', 'dump initial execution graph to the file specified') { |o|
+    options[:gdump] = o
+  }
+  opts.on('-t', '--timeout SECONDS', 'for how long to wait for the vertices to execute') { |o|
+    options[:timeout] = o.to_i
+  }
+}.parse!
+
+logfile, ui_drawer =
+  case options[:logfile]
+  when ':stdout', $stdout, :stdout
+    [$stdout, nil]
+  when ':silent'
+    [nil, nil]
+  else # all other strings are considered log file names
+    [options[:logfile], ExcADG::Tui]
+  end
+
+if logfile.nil?
+  ExcADG::Log.mute
+else
+  ExcADG::Log.logger ExcADG::Log::RLogger.new dest: logfile, level: options[:loglevel]
+end
+
+raise "'#{options[:graph]}' config file is not readable" unless File.readable? options[:graph]
+
+ExcADG::Log.info 'reading config'
+config = YAML.safe_load_file options[:graph]
+
+runnable_vertices = config.select { |k, v| v&.dig('dependencies').nil? }.keys
+raise ArgumentError, 'at least one vertex should be ready to start' if runnable_vertices.empty?
+
+ExcADG::Log.info 'collect execution graph from config'
+
+all_vertex_names = config.keys.collect!(&:to_sym)
+graph = RGL::DirectedAdjacencyGraph.new
+config.each_pair { |id, vconfig|
+  name = id.to_sym
+  payload = if vconfig&.key?('command')
+              ExcADG::Payload::Wrapper::Bin.new args: vconfig['command']
+            elsif vconfig&.key?('sleep')
+              ExcADG::Payload::Example::Sleepy.new args: vconfig['sleep'].to_i
+            else
+              ExcADG::Payload::Example::Echo.new args: vconfig
+            end
+  deps_v_names = (vconfig&.dig('dependencies') || []).collect(&:to_sym)
+  deps_v_names.each { |dep|
+    raise "there is no dependency named '#{dep}' in the graph for vertex '#{id}'" unless all_vertex_names.include? dep
+
+    graph.add_edge name, dep
+  }
+
+  ExcADG::Vertex.new name:, payload:, deps: deps_v_names
+}
+
+graph.write_to_graphic_file(options[:gdump].split('.').last, options[:gdump].split('.')[...-1].join('.')) unless options[:gdump].nil?
+
+ExcADG::Log.info 'starting state data broker'
+ExcADG::Broker.run
+
+ui_drawer&.run
+
+ExcADG::Log.info 'watching for all vertices to complete'
+timed_out = false
+begin
+  ExcADG::Broker.wait_all(timeout: options[:timeout]).join
+rescue Timeout::Error
+  ExcADG::Log.error 'execution timed out'
+  timed_out = true
+end
+vertice_by_state = ExcADG::Broker.data_store.to_a.group_by(&:state)
+
+ExcADG::Log.info "#{vertice_by_state[:done]&.size || 0} done, #{vertice_by_state[:failed]&.size || 0} failed"
+has_failed = !vertice_by_state[:failed].nil?
+ui_drawer&.summarize has_failed, timed_out
+
+unless options[:dump].nil?
+  ExcADG::Log.info "writing data to #{options[:dump]}"
+  File.open(options[:dump], 'w+') { |f|
+    f.write JSON.dump ExcADG::Broker.data_store.to_a
+  }
+end
+
+sleep 0.1 # let the logger to print message
+exit_code = 0
+exit_code |= 1 if has_failed
+exit_code |= 2 if timed_out
+exit exit_code

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: excadg
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.1.2
 platform: ruby
 authors:
 - skorobogatydmitry
@@ -10,15 +10,17 @@ bindir: bin
 cert_chain: []
 date: 2024-07-06 00:00:00.000000000 Z
 dependencies: []
-description: "# Description\n\nThat's a library (framework) to execute a graph of
-  dependent tasks (vertices).  \nIts main feature is to run all possible tasks independently
-  in parallel once they're ready.  \nAnother feature is that the graph is dynamic
-  and any vertex could produce another vertice(s) to extend execution graph.\n"
+description: "\n\nThat's a library (framework) to execute a graph of dependent tasks
+  (vertices).  \nIts main feature is to run all possible tasks independently in parallel
+  once they're ready.  \nAnother feature is that the graph is dynamic and any vertex
+  could produce another vertice(s) to extend execution graph.\n"
 email: skorobogaty.dmitry@gmail.com
 executables: []
 extensions: []
 extra_rdoc_files: []
 files:
+- README.md
+- bin/excadg
 - lib/excadg.rb
 - lib/excadg/assertions.rb
 - lib/excadg/broker.rb