rsmp 0.48.4 → 0.51.0

data/documentation/tasks.md

@@ -1,148 +1,96 @@
-# Tasks
+# Async task ownership
 
-## Concurrency
-When you use a site or a supervisor, it runs asyncronously so you can run several concurrently, or do other things concurrently, like sending messages and waiting for reponses.
+RSMP uses Ruby's fiber scheduler and the Async gem for connections, readers,
+timers, and waits. A site, supervisor, or proxy is not itself an Async task. It
+owns a main task exposed through `task`.
 
-Concurrency is provided by Ruby Fiber Scheduler and the Async gem. and used to:
+## Starting and waiting
 
-- Runn multiple sites or supervisors
-- Run multiple connections from each site or supervisor
-- Wait for messages, connections or states
-
-Classes don't inherit from Async::Task. Instead they include task as instance variables. This means that the hierachy of objects and tasks can be different.
-
-## Task module
-The `Task` module is used by the `Node` and `Proxy` classes and defines a life cycle for handling async tasks.
-
-A single main tasks is kept in `@task`. If subclasses need subtask, they can start them as needed, inside the main task.
-
-You first call `start`. If `@task` already exists, it will return immedatiately.
-Otherwise an async task is created and stored in `@task`, and `run` is called inside this task, to handle any long-running jobs, like listening for incoming messages. By default `run` calls `start_subtasks`, but should be overriden to do actual work as well.
-The call to `start` returns immediately, with the async task running concurently.
-
-If you want to stop the task, call `stop`. It will fist call `stop_subtasks` and then `stop_task` which calls Async#stop on `@task` and sets `@task` to `nil`.
-
-## Stopping tasks
-If a task stops itself, code after the call to `stop` will not be run, unless you use an ensure block:
+`start` must run below an active Async parent. It returns the created
+`Async::Task`; it never creates a hidden root reactor.
 
 ```ruby
-require 'async'
-
 Async do |task|
-	task.stop
-	puts "I just stopped"   # this will not be reaced, because the task was stopped
-end
+  site = RSMP::Site.new(site_settings: settings)
+  site.start(parent: task)
 
-Async do |task|
-	task.stop
-ensure
-	puts "I just stopped"   # this will be reached
+  # Other work can run here.
+  site.stop
 end
 ```
-This can happen if either the task stops itself directly, or if it stops task higher up in the task hierarchy.
-
-For example, when a timer task finds that an acknowledgement was not received in time, and then closes the connection by calling stop() on the Proxy, the proxy which will then in turn stop the timer task. So any code that should be run after that must be inside an ensure block.
-
-## Proxies
-A site connects to one of more supervisors. A proxy is created and run to handle each connection.
-
-A site can connect to one or more supervisor. It creates a proxy for each and runs them.
-
-When the proxy is run, it creates an async task to read incoming messages. This reader task is stored in `@reader`and will be a sub task of the site/supervisor main task.
-
-A proxy also creates a timer task, stored in `@timer`. This task is used to check watchdog and acknowledgement timeouts.
-
-Proxies build on the Task module functionality by handling RSMP communication via a TCP socket.
-
-The TCP socket can be open or closed. The RSMP communication first goes through a handshake sequence before being ready. The `status` attribute keeps track of this, and can be either `:disconnected`, `:connected` or `:ready`
-
-Proxies provide `connect` and `close` for starting and stopping commununication.
 
-A supervisor proxies connects actively to a site proxy, whereas a site proxy waits for the supervisor proxy to connect.
+Inside an Async task, the current task is the default parent, so `site.start` is
+equivalent. Calling `start` again while the task is running returns the existing
+task.
 
-A supervisor proxy is created at startup, and is responsible for creating the TCP socket and connecting to the supervisor.
+`wait` uses `Async::Task#wait`. If the task ends because of an unexpected
+exception, `wait` raises that same exception with its original backtrace.
+Cancellation is handled by Async and is not translated into an operational
+failure.
 
-A site proxy is also created at startup, but the socket is created in the supervisor by `Aync::Endpoint#accept`when a site connects.
+## Structured task trees
 
-## Object Hierarchy
-A Site has one or more SupervisorProxies, which represent connections to the remote supervisor. 
+Every long-lived child has an explicit owner:
 
-A Supervisor has one or more SiteProxies, which represent connections to the remote site.
-
-SiteProxy and SupervisorProxy both inherit from Proxy and have a reader task and a timer task. These are just Async tasks, not separate classes.
-
-## Task Hierachy
-When you start an Async task, the parent will be the currently running task, unless you specify a different parent task.
-
-The Async task hierarchy is similar to the object hierachy, with the difference that proxies have reader and timer tasks. And IO::Endpoint which is used to handle TCP connections concurently, will create some intermediate tasks:
-
-```
-Supervisor @task
-	accepting connections      # this task is created by IO::Endpoint
-		incoming connection 1    # this task is created by IO::Endpoint
-			SiteProxy @task
-				SiteProxy @reader
-				SiteProxy @timer
-		incoming connection 2    # this task is created by IO::Endpoint
-			SiteProxy @task
-				SiteProxy @reader
-				SiteProxy @timer
+```text
+Site or Supervisor main task
+├── listener or connection proxy tasks
+├── site status timer
+└── explicit termination waiter
 
+Proxy main task
+├── connection reader
+└── connection timer
 ```
 
+Sibling tasks are registered in an `Async::Barrier`. Their owner observes task
+completion in completion order. A reader's expected EOF or transport failure is
+returned as an `RSMP::Result`, closes that connection session, and can lead to a
+reconnect. An exception from message processing or a timer is not caught as a
+connection error: it propagates through the barrier to the proxy and then the
+node owner.
 
-A Site run from the CLI, before it connects to a supervisor:
+Session IDs prevent a late task from an old connection from closing or
+publishing termination for a newer connection. Closing is idempotent and
+cancels the remaining reader/timer sibling.
 
-```
-#<Async::Reactor:0xb90 1 children (running)>
-	#<Async::Task:0xba4 cli (running)>
-		#<Async::Task:0xbcc RSMP::TLC::TrafficControllerSite main task (running)>
-			#<Async::Task:0xbe0 RSMP::SupervisorProxy main task (running)>
-			#<Async::Task:0xbf4 tlc timer (running)>
-```
+## Finite waits
 
-After the site connects to a supervisor:
+Finite waits return `RSMP::Result`:
 
+```ruby
+result = supervisor.wait_for_site('RN+SI0001', timeout: 5)
+result = proxy.wait_for_state(:ready, timeout: 5)
 ```
-#<Async::Reactor:0xadc 1 children (running)>
-	#<Async::Task:0xaf0 cli (running)>
-		#<Async::Task:0xb18 RSMP::TLC::TrafficControllerSite main task (running)>
-			#<Async::Task:0xb2c RSMP::SupervisorProxy main task (running)>
-				#<Async::Task:0xb40 reader (running)>
-				#<Async::Task:0xb54 timer (running)>
-			#<Async::Task:0xb68 tlc timer (running)>
-```
-
 
+A normal timeout is `Result::Failure` with code `:timeout`. Use
+`wait_for_site!`, `wait_for_state!`, or another bang variant to raise
+`RSMP::OperationError` explicitly.
 
-A supervisor run from the CLI, before any sites have connected:
+## Events from long-running services
 
-```
-#<Async::Reactor:0x94c 1 children (running)>
-	#<Async::Task:0x960 cli (running)>
-		#<Async::Task:0x988 RSMP::Supervisor main task (running)>
-			#<Async::Task:0x99c accepting connections #<Addrinfo: 0.0.0.0:14111 TCP> [fd=12] (running)>
-```
+Events report occurrences that are not the result of a single finite call, such
+as a failed connection attempt, invalid peer message, missing acknowledgement,
+missing watchdog, or connection end. Register an explicit subscriber on the
+node:
 
-The supervisor after a site has connected
+```ruby
+receiver = Object.new
+receiver.define_singleton_method(:receive_event) do |event|
+  warn "#{event.type}: #{event.failure}"
+end
 
+site.add_event_receiver(receiver)
 ```
-#<Async::Reactor:0x94c 1 children (running)>
-	#<Async::Task:0x960 cli (running)>
-		#<Async::Task:0x988 RSMP::Supervisor main task (running)>
-			#<Async::Task:0x99c accepting connections #<Addrinfo: 0.0.0.0:13111 TCP> [fd=12] (running)>
-				#<Async::Task:0xac8 incoming connection #<Addrinfo: 127.0.0.1:51778 TCP> [fd=13] (running)>
-					#<Async::Task:0xadc RSMP::SiteProxy main task (running)>
-						#<Async::Task:0xaf0 reader (running)>
-						#<Async::Task:0xb04 timer (running)>
-```
-
-The task hierachy matters when you stop a task or iterate on subtasks. Note that calling `Task#wait`
- does not wait for subtasks, whereas Task#stop stops all subtasks.
 
-## Transient tasks
-If you mark an Async task with `transient: true` when you created it, that task will be stopped aas soon as all normal task are completed. They will also be moved up the hierarchy if the parent is complete, but not the grandparent.
+Events are ordered and delivered in the reactor. There is no implicit global
+error queue. Subscribers must have an owner that drains or handles them. A
+subscriber callback defect propagates unless the subscriber implements `crash`,
+in which case it receives that exception through its own completion mechanism.
 
-Transient tasks can be used to cleanup, but using an `ensure` block in the transient task. When you call `stop` on a task, an Async::Stop exception is raised, which will run the code in the `ensure` block.
+## Restart and shutdown
 
-Some sus tests run in transient tasks. As soon as the main test code is complete, any subtasks like Sites or Supervisors that might otherwise prevent the test from completing will be stopped automatically.
+`stop` cancels the owned main task and its descendants. A TLC restart request is
+not an exception: it resolves an explicit `RSMP::Termination` with reason
+`:restart`. The CLI observes this value, lets the old structured task tree clean
+up, and then constructs a new site.

data/lib/rsmp/cli.rb

@@ -21,6 +21,91 @@ module RSMP
     end
   end
 
+  # CLI subcommands for SXL dependency resolution and manifests.
+  class SXLCLI < Thor
+    namespace :sxl
+
+    desc 'resolve ROOT...', 'Resolve SXL dependencies and write a manifest'
+    method_option :source, type: :array, aliases: '-s', default: [],
+                           banner: 'Local SXL file or directory paths'
+    method_option :out, type: :string, aliases: '-o', default: 'manifest.yaml',
+                        banner: 'Manifest output path'
+    method_option :format, type: :string, default: RSMP::Schema.latest_core_version,
+                           enum: RSMP::Schema.core_versions,
+                           banner: 'RSMP Core manifest format version'
+    method_option :force, type: :boolean, aliases: '-f', default: false,
+                          banner: 'Overwrite an existing manifest'
+    def resolve(*roots)
+      raise RSMP::SXL::Processing::Error, 'sxl resolve requires at least one root' if roots.empty?
+
+      ensure_manifest_output_available!
+      catalogue = RSMP::SXL::Processing::Catalogue.new(options[:source])
+      requirements = roots.map { |root| root_requirement(root, catalogue) }
+      documents = RSMP::SXL::Processing::Resolver.new(catalogue).resolve(requirements)
+      manifest = RSMP::SXL::Processing::Manifest.create(documents, format: options[:format])
+      write_manifest(manifest)
+      puts "Wrote #{options[:out]}"
+    rescue RSMP::SXL::Processing::Error, SystemCallError => e
+      fail_sxl_command(e)
+    end
+
+    desc 'verify [MANIFEST]', 'Verify an SXL manifest against local sources'
+    method_option :source, type: :array, aliases: '-s', default: [],
+                           banner: 'Local SXL file or directory paths'
+    def verify(path = 'manifest.yaml')
+      catalogue = RSMP::SXL::Processing::Catalogue.new(options[:source])
+      manifest = RSMP::SXL::Processing::Manifest.load(path)
+      RSMP::SXL::Processing::Manifest.verify!(manifest, catalogue)
+      puts 'OK'
+    rescue RSMP::SXL::Processing::Error, SystemCallError => e
+      fail_sxl_command(e)
+    end
+
+    private
+
+    def root_requirement(root, catalogue)
+      return path_root_requirement(root, catalogue) if File.exist?(root)
+
+      name, requirement = root.split(':', 2)
+      unless requirement
+        raise RSMP::SXL::Processing::Error,
+              "Invalid root #{root.inspect}; expected an SXL YAML path or name:requirement"
+      end
+
+      RSMP::SXL::Processing::Document.validate_name!(name)
+      [name, RSMP::SXL::Processing::VersionRequirement.new(requirement)]
+    end
+
+    def path_root_requirement(path, catalogue)
+      raise RSMP::SXL::Processing::Error, "Root SXL #{path} is not a file" unless File.file?(path)
+
+      document = RSMP::SXL::Processing::Document.load(path)
+      catalogue.add_document(document)
+      [document.name, RSMP::SXL::Processing::VersionRequirement.new(document.version_string)]
+    end
+
+    def ensure_manifest_output_available!
+      output = options[:out]
+      return unless File.exist?(output)
+      raise RSMP::SXL::Processing::Error, "Manifest output #{output} is a directory" if File.directory?(output)
+      return if options[:force]
+
+      raise RSMP::SXL::Processing::Error,
+            "Refusing to overwrite existing manifest #{output}; use --force to replace it"
+    end
+
+    def write_manifest(manifest)
+      output = options[:out]
+      FileUtils.mkdir_p(File.dirname(File.expand_path(output)))
+      File.write(output, RSMP::SXL::Processing::Manifest.dump(manifest))
+    end
+
+    def fail_sxl_command(error)
+      puts "Error: #{error.message}"
+      exit 1
+    end
+  end
+
   # CLI subcommands for RSMP configuration validation.
   class ConfigCLI < Thor
     namespace :config
@@ -78,8 +163,9 @@ module RSMP
       puts "Error: #{e}"
       exit 1
     rescue StandardError => e
-      puts "Uncaught error: #{e}"
-      puts caller.join("\n")
+      warn "Uncaught error: #{e.class}: #{e.message}"
+      warn e.full_message(highlight: false, order: :top)
+      exit 1
     end
 
     desc 'supervisor', 'Run RSMP supervisor'
@@ -100,6 +186,7 @@ module RSMP
     end
 
     register SchemaCLI, 'schema', 'schema COMMAND', 'SXL schema commands'
+    register SXLCLI, 'sxl', 'sxl COMMAND', 'SXL dependency and manifest commands'
     register ConfigCLI, 'config', 'config COMMAND', 'Configuration commands'
 
     private
@@ -194,7 +281,10 @@ module RSMP
         loop do
           site = site_class.new(site_settings: settings, log_settings: log_settings)
           site.start
-          site.wait
+          termination = site.wait
+          break unless termination&.success? && termination.value.reason == :restart
+
+          site.stop
         rescue Psych::SyntaxError => e
           puts "Cannot read config file #{e}"
           break
@@ -202,8 +292,6 @@ module RSMP
                RSMP::ConfigurationError => e
           puts "Cannot start site: #{e}"
           break
-        rescue RSMP::Restart
-          site.stop
         end
       end
     end