glazed 0.1.0 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (50) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +19 -0
  3. data/lib/glazed/collector.rb +10 -6
  4. data/lib/glazed/command/base.rb +3 -18
  5. data/lib/glazed/command/orchestrate.rb +1 -1
  6. data/lib/glazed/command/run.rb +1 -1
  7. data/lib/glazed/command/send_signal.rb +21 -13
  8. data/lib/glazed/config.rb +50 -0
  9. data/lib/glazed/coordinator/task_runner.rb +189 -0
  10. data/lib/glazed/coordinator.rb +252 -0
  11. data/lib/glazed/dialect/run.rb +9 -3
  12. data/lib/glazed/dialect/sensitive_pattern.rb +7 -2
  13. data/lib/glazed/orchestrator/policy.rb +28 -0
  14. data/lib/glazed/orchestrator.rb +15 -65
  15. data/lib/glazed/proto.rb +1 -0
  16. data/lib/glazed/redactor.rb +65 -26
  17. data/lib/glazed/secret.rb +12 -37
  18. data/lib/glazed/secrets/command.rb +66 -0
  19. data/lib/glazed/secrets.rb +27 -5
  20. data/lib/glazed/secrets_resolver.rb +85 -0
  21. data/lib/glazed/shell_command.rb +41 -6
  22. data/lib/glazed/sources/command.rb +11 -8
  23. data/lib/glazed/state_backends/base.rb +11 -1
  24. data/lib/glazed/state_backends/sqlite.rb +38 -1
  25. data/lib/glazed/step_context.rb +26 -8
  26. data/lib/glazed/steps/{batch_shell.rb → batch_command.rb} +15 -15
  27. data/lib/glazed/steps/{shell.rb → command.rb} +21 -18
  28. data/lib/glazed/steps/fetch.rb +1 -1
  29. data/lib/glazed/steps/redact.rb +3 -1
  30. data/lib/glazed/storage_context.rb +22 -0
  31. data/lib/glazed/storages/base.rb +3 -2
  32. data/lib/glazed/storages/filesystem.rb +3 -2
  33. data/lib/glazed/storages/git.rb +25 -8
  34. data/lib/glazed/storages/github.rb +22 -8
  35. data/lib/glazed/storages/memory.rb +3 -2
  36. data/lib/glazed/storages.rb +6 -4
  37. data/lib/glazed/transports/base.rb +14 -2
  38. data/lib/glazed/transports/ssh.rb +55 -2
  39. data/lib/glazed/transports/telnet.rb +4 -2
  40. data/lib/glazed/v1/config_pb.rb +2 -1
  41. data/lib/glazed/v1/coordinator_pb.rb +34 -0
  42. data/lib/glazed/v1/inventory_pb.rb +5 -1
  43. data/lib/glazed/v1/orchestrator_pb.rb +1 -1
  44. data/lib/glazed/v1/redaction_pb.rb +2 -1
  45. data/lib/glazed/v1/secret_pb.rb +3 -1
  46. data/lib/glazed/v1/state_pb.rb +2 -1
  47. data/lib/glazed/v1/storage_pb.rb +1 -1
  48. data/lib/glazed/v1/workflow_pb.rb +3 -3
  49. data/lib/glazed/version.rb +1 -1
  50. metadata +10 -3
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: fc8ac8baaf98756ff8b05fd712251274f27ec1c93bd02b980c528a5e8e059970
4
- data.tar.gz: db6f2becb3b6d74ad1e44f8f8f385641e9732e92d54cf536130150988bac395b
3
+ metadata.gz: f09d16d28a4ecb996fa05ba53cc4af10d5cbe45b65c09f3830f88326a6e200c5
4
+ data.tar.gz: 7c40270ad31553fac8a681ba5369c1596cfaed4a8c09cf66036115def90bd4d9
5
5
  SHA512:
6
- metadata.gz: 64eea926af805078a1e5e0a4653839e2c5ef5207f5515d1a8bfeb464f8ff2dbfda58ca6d25dc596c179ae8e9594f793f1548e10606b0bd2005ca087a735fc6c9
7
- data.tar.gz: fe504897bf58461f509b0ca08e619661c52e04a9c75d61f4349d97d98679cb0f95ccdce73ad5216adfb7b1492f40f886bc14fc137c6ec11bf86be2982aacd41d
6
+ metadata.gz: d12cde4376d07ae82221202ebab2ec22a473b85dcdae0e2a5352d2f82235ed8e059a7a66fc7266d1d6d5e14eac5d68b2fdc05a34b22991aac18111d1620bd21e
7
+ data.tar.gz: b0e2f351f6ff580fd4b639162088ad4bfd3d947d8591805453129703ed227569d94c605124d43d760824d650c318450b6ab93766886c5e1fc2cd859b4eec1373
data/CHANGELOG.md CHANGED
@@ -1,3 +1,22 @@
1
+ ## [0.2.0] - 2026-10-15
2
+
3
+ ### Breaking changes
4
+
5
+ - Secrets are read through a `Glazed::SecretsResolver`, which `StepContext` holds and passes to the collector, dialect runs, the redactor and storages. `Glazed::Secret.new` takes the resolver, and `Secret.maybe` is replaced by `SecretsResolver#maybe`
6
+ - Storages are built with `from_proto(settings, context:)`, where a `Glazed::StorageContext` carries the secrets resolver; plugin storages must accept it
7
+ - The `shell` and `batch_shell` steps are renamed `command` and `batch_command` (`glazed.v1.CommandStep` and `BatchCommandStep`), as they also run programs without a shell
8
+
9
+ ### Enhancements
10
+
11
+ - Redaction hides additional secrets given as secret references, as literal text or regular expressions and optionally split by a delimiter, in a `redact` step or in the top-level `redaction` shared by all of them
12
+ - Storages: `subdirectory` for `git` and `github` places files in a directory of the repository instead of its root
13
+ - `glazed signal --all`, and `all_nodes` and `workflows` on `glazed.v1.Signal`, signal every node of chosen workflows, for scheduled runs
14
+ - `Glazed::Coordinator` and the `glazed.v1.CoordinatorTask` protocol let an external coordinator, such as AWS Step Functions with glazed-aws, drive the orchestrated workflows one step at a time
15
+ - `SecretRef` gains `ssm`, an AWS Systems Manager parameter read by glazed-aws
16
+ - `SecretRef` gains `command`, a value printed by a command
17
+ - `command` and `batch_command` steps, `command` secrets and the `command` inventory take `exec`, a program and its arguments executed without a shell, in place of `run`
18
+ - `sqlite` state backend: schema version 2 keeps the waiting coordinator execution. Version 1 databases are upgraded on first use, after which glazed 0.1.0 refuses them
19
+
1
20
  ## [0.1.0] - 2026-10-14
2
21
 
3
22
  Initial release.
@@ -8,7 +8,7 @@ require 'glazed/cli_session'
8
8
  require 'glazed/dialect/run'
9
9
  require 'glazed/dialects'
10
10
  require 'glazed/proto'
11
- require 'glazed/secret'
11
+ require 'glazed/secrets_resolver'
12
12
  require 'glazed/transports'
13
13
 
14
14
  module Glazed
@@ -25,8 +25,10 @@ module Glazed
25
25
  # @param transport [Transports::Base, nil] transport to use instead of building one from the node
26
26
  # @param timeout [Numeric] seconds to wait for each expected output
27
27
  # @param progress [Proc] receives a message when connecting and whenever waiting for the device
28
- #: (node: Proto::V1::Node, ?transport: Transports::Base?, ?timeout: Numeric, ?progress: ^(String) -> void) -> void
29
- def initialize(node:, transport: nil, timeout: CliSession::DEFAULT_TIMEOUT, progress: ->(_message) {})
28
+ # @param secrets [SecretsResolver] reads the node's password and secrets
29
+ #: (node: Proto::V1::Node, ?transport: Transports::Base?, ?timeout: Numeric, ?progress: ^(String) -> void, ?secrets: SecretsResolver) -> void
30
+ def initialize(node:, transport: nil, timeout: CliSession::DEFAULT_TIMEOUT, progress: ->(_message) {}, secrets: SecretsResolver.new)
31
+ @secrets = secrets #: SecretsResolver
30
32
  @node = node #: Proto::V1::Node
31
33
  @dialect = Dialects.find(node.dialect) #: singleton(Dialects::Base)
32
34
  @kind = (node.transport.empty? ? DEFAULT_TRANSPORT : node.transport) #: String
@@ -60,7 +62,7 @@ module Glazed
60
62
  timeout: @timeout,
61
63
  progress: @progress,
62
64
  )
63
- run = Dialect::Run.new(cli: session, node:, dialect:)
65
+ run = Dialect::Run.new(cli: session, node:, dialect:, secrets: @secrets)
64
66
  @transport.pty_options = @config.pty_options
65
67
  @progress.call("connecting to #{@transport.host} over #{@kind}")
66
68
  @transport.open
@@ -93,7 +95,7 @@ module Glazed
93
95
  end
94
96
  if (password_prompt = @config.password_prompt)
95
97
  session.expect(password_prompt)
96
- session.write("#{Secret.maybe(node.password)&.value}\n")
98
+ session.write("#{@secrets.maybe(node.password)&.value}\n")
97
99
  end
98
100
  session.expect(session.prompt)
99
101
  end
@@ -106,7 +108,9 @@ module Glazed
106
108
  host: node.host.empty? ? node.name : node.host,
107
109
  port: node.port.zero? ? nil : node.port,
108
110
  username: node.username,
109
- password: Secret.maybe(node.password),
111
+ password: @secrets.maybe(node.password),
112
+ transport_options: node.transport_options,
113
+ secrets: @secrets,
110
114
  )
111
115
  end
112
116
  end
@@ -6,13 +6,12 @@
6
6
  require 'json'
7
7
  require 'dry/cli'
8
8
  require 'glazed'
9
+ require 'glazed/config'
9
10
  require 'glazed/must'
10
11
  require 'glazed/proto'
11
- require 'glazed/proto_json'
12
12
  require 'glazed/runner'
13
13
  require 'glazed/state_backends'
14
14
  require 'glazed/step_context'
15
- require 'glazed/validation'
16
15
 
17
16
  module Glazed
18
17
  module Command
@@ -91,8 +90,7 @@ module Glazed
91
90
  #: (Proto::V1::Config config) -> Integer
92
91
  private def default_jobs(config) = config.jobs.zero? ? DEFAULT_JOBS : config.jobs
93
92
 
94
- # Reads the config, first requiring the features its `requires` lists so
95
- # plugin settings types are known when it is decoded.
93
+ # Reads the config with {Glazed::Config.load}.
96
94
  #
97
95
  # @param path [String] config file
98
96
  # @return [Proto::V1::Config] the config
@@ -100,20 +98,7 @@ module Glazed
100
98
  # @raise [InvalidMessage] when the config breaks the rules of its schema
101
99
  # @raise [LoadError] when a required feature is missing
102
100
  #: (String path) -> Proto::V1::Config
103
- private def load_config(path)
104
- json = ::File.read(path)
105
- document = JSON.parse(json) #: untyped
106
- requires = document.is_a?(Hash) ? document.fetch('requires', []) : []
107
- unless requires.is_a?(Array) && requires.all?(String)
108
- raise Glazed::Error, "#{path}: requires must be a list of strings"
109
- end
110
-
111
- requires.each { require it }
112
- ProtoJson.load(Proto::V1::Config, json).tap { Validation.validate!(it, origin: path) }
113
- rescue Google::Protobuf::ParseError, JSON::ParserError => e
114
- hint = e.message.include?('Type was not found') ? '; list the gem providing that plugin type in requires' : ''
115
- raise Glazed::Error, "#{path}: #{e.message}#{hint}"
116
- end
101
+ private def load_config(path) = Glazed::Config.load(path)
117
102
 
118
103
  # @param config [Proto::V1::Config] the config
119
104
  # @param name [String, nil] requested workflow
@@ -38,7 +38,7 @@ module Glazed
38
38
  settings = load_config(config)
39
39
  orchestration = orchestrator_settings(settings, config)
40
40
  listener = json_listener
41
- context = StepContext.from_proto(settings.storages, listener:)
41
+ context = StepContext.from_proto(settings.storages, redaction: settings.redaction, listener:)
42
42
  workflows = orchestration.workflows.keys.to_h { [it, Workflow.from_proto(select_workflow(settings, it), context:)] }
43
43
  backend = state_backend(orchestration, context)
44
44
  orchestrator = Orchestrator.new(
@@ -37,7 +37,7 @@ module Glazed
37
37
 
38
38
  settings = load_config(config)
39
39
  listener = json_listener
40
- context = StepContext.from_proto(settings.storages, listener:)
40
+ context = StepContext.from_proto(settings.storages, redaction: settings.redaction, listener:)
41
41
  steps = Workflow.from_proto(select_workflow(settings, workflow), context:)
42
42
  nodes = select_nodes(Sources.nodes(settings.inventory.to_a), only || [])
43
43
 
@@ -6,8 +6,8 @@
6
6
  require 'time'
7
7
  require 'google/protobuf'
8
8
  require 'glazed/command/base'
9
+ require 'glazed/coordinator'
9
10
  require 'glazed/must'
10
- require 'glazed/orchestrator'
11
11
  require 'glazed/proto'
12
12
  require 'glazed/sources'
13
13
  require 'glazed/state_backends'
@@ -23,42 +23,50 @@ module Glazed
23
23
  # each node is due.
24
24
  class SendSignal < Base
25
25
  desc 'Make nodes due soon for glazed orchestrate'
26
- argument :nodes, type: :array, required: true, desc: 'Names of the nodes'
26
+ argument :nodes, type: :array, required: false, desc: 'Names of the nodes'
27
27
  option :config, aliases: ['-c'], desc: 'Config file, ProtoJSON of glazed.v1.Config; $GLAZED_CONFIG_FILE when omitted'
28
28
  option :workflow, aliases: ['-w'], type: :array, desc: 'Signal only these orchestrated workflows, comma-separated'
29
+ option :all, type: :boolean, default: false, desc: 'Signal every node in the inventory instead of the named ones'
29
30
  option :source, default: 'manual', desc: 'What sent the signal, recorded as the reason the node is due'
30
31
  option :now, type: :boolean, default: false, desc: 'Make the nodes due now, without the debounce delay'
31
32
 
32
33
  # Records the signals.
33
34
  #
34
- # @param nodes [Array<String>] node names
35
+ # @param nodes [Array<String>, nil] node names; none with all
35
36
  # @param config [String, nil] path of the config file; {Base::CONFIG_FILE_ENV} when nil
36
37
  # @param workflow [Array<String>, nil] workflows to signal; every orchestrated one when nil
38
+ # @param all [Boolean] whether to signal every node in the inventory
37
39
  # @param source [String] what sent the signal
38
40
  # @param now [Boolean] whether to skip the debounce delay
39
41
  # @param _options [Hash] other options dry-cli passes, unused
40
42
  # @return [void]
41
43
  # @raise [Glazed::Error, Google::Protobuf::ParseError] when the config is invalid, a node is unknown, or the state backend fails
42
- #: (nodes: Array[String], ?config: String?, ?workflow: Array[String]?, ?source: String, ?now: bool, **untyped _options) -> void
43
- def call(nodes:, config: nil, workflow: nil, source: 'manual', now: false, **_options)
44
+ #: (?nodes: Array[String]?, ?config: String?, ?workflow: Array[String]?, ?all: bool, ?source: String, ?now: bool, **untyped _options) -> void
45
+ def call(nodes: nil, config: nil, workflow: nil, all: false, source: 'manual', now: false, **_options)
44
46
  config = config_path(config, 'signal')
47
+ nodes ||= []
48
+ if all != nodes.empty?
49
+ err.puts 'glazed signal: name nodes, or give --all'
50
+ exit 1
51
+ end
45
52
 
46
53
  settings = load_config(config)
47
54
  orchestration = orchestrator_settings(settings, config)
48
55
  workflows = select_orchestrated(orchestration, workflow)
49
- select_nodes(Sources.nodes(settings.inventory.to_a), nodes)
56
+ inventory = Sources.nodes(settings.inventory.to_a).map(&:name)
57
+ select_nodes(inventory.map { Proto::V1::Node.new(name: it) }, nodes)
50
58
  context = StepContext.from_proto(settings.storages, listener: json_listener)
51
59
  backend = state_backend(orchestration, context)
52
60
  at = Time.now
53
- signals = nodes.map do |node|
54
- Proto::V1::Signal.new(node:, source:, received_at: Google::Protobuf::Timestamp.from_time(at), immediate: now).tap { Validation.validate!(it) }
61
+ received_at = Google::Protobuf::Timestamp.from_time(at)
62
+ signals = (all ? [nil] : nodes.uniq).map do |node|
63
+ Proto::V1::Signal.new(node: node || '', all_nodes: all, source:, received_at:, immediate: now, workflows:).tap { Validation.validate!(it) }
55
64
  end
65
+ records = Coordinator.new(settings: orchestration, backend:, clock: -> { at }).signal(signals, nodes: inventory)
56
66
  workflows.each do |name|
57
- debounce = orchestration.workflows[name]&.debounce
58
- records = Orchestrator.signal(backend:, workflow: name, signals:, debounce:, now: at)
59
- nodes.uniq.each do |node|
60
- due_at = Must.because(records.fetch(node).due_at) { 'validation requires due_at' }
61
- out.puts "#{node} #{name} due at #{due_at.to_time.utc.iso8601}"
67
+ records.fetch(name, {}).each_value do |record|
68
+ due_at = Must.because(record.due_at) { 'validation requires due_at' }
69
+ out.puts "#{record.node} #{name} due at #{due_at.to_time.utc.iso8601}"
62
70
  end
63
71
  end
64
72
  ensure
@@ -0,0 +1,50 @@
1
+ # typed: strict
2
+ # frozen_string_literal: true
3
+
4
+ # SPDX-License-Identifier: Apache-2.0
5
+
6
+ require 'json'
7
+ require 'google/protobuf'
8
+ require 'glazed'
9
+ require 'glazed/proto'
10
+ require 'glazed/proto_json'
11
+ require 'glazed/validation'
12
+
13
+ module Glazed
14
+ # Reads glazed configs ({Proto::V1::Config}).
15
+ module Config
16
+ # Reads a config file with {.parse}.
17
+ #
18
+ # @param path [String] config file
19
+ # @return [Proto::V1::Config] the config
20
+ # @raise [Glazed::Error] when `requires` is malformed, or the config does not decode
21
+ # @raise [InvalidMessage] when the config breaks the rules of its schema
22
+ # @raise [LoadError] when a required feature is missing
23
+ #: (String path) -> Proto::V1::Config
24
+ def self.load(path) = parse(::File.read(path), origin: path)
25
+
26
+ # Decodes a config, first requiring the features its `requires` lists so
27
+ # plugin settings types are known when it is decoded.
28
+ #
29
+ # @param json [String] ProtoJSON of {Proto::V1::Config}
30
+ # @param origin [String] where the config comes from, for error messages
31
+ # @return [Proto::V1::Config] the config
32
+ # @raise [Glazed::Error] when `requires` is malformed, or the config does not decode
33
+ # @raise [InvalidMessage] when the config breaks the rules of its schema
34
+ # @raise [LoadError] when a required feature is missing
35
+ #: (String json, origin: String) -> Proto::V1::Config
36
+ def self.parse(json, origin:)
37
+ document = JSON.parse(json) #: untyped
38
+ requires = document.is_a?(Hash) ? document.fetch('requires', []) : []
39
+ unless requires.is_a?(Array) && requires.all?(String)
40
+ raise Glazed::Error, "#{origin}: requires must be a list of strings"
41
+ end
42
+
43
+ requires.each { require it }
44
+ ProtoJson.load(Proto::V1::Config, json).tap { Validation.validate!(it, origin:) }
45
+ rescue Google::Protobuf::ParseError, JSON::ParserError => e
46
+ hint = e.message.include?('Type was not found') ? '; list the gem providing that plugin type in requires' : ''
47
+ raise Glazed::Error, "#{origin}: #{e.message}#{hint}"
48
+ end
49
+ end
50
+ end
@@ -0,0 +1,189 @@
1
+ # typed: strict
2
+ # frozen_string_literal: true
3
+
4
+ # SPDX-License-Identifier: Apache-2.0
5
+
6
+ require 'glazed'
7
+ require 'glazed/coordinator'
8
+ require 'glazed/must'
9
+ require 'glazed/proto'
10
+ require 'glazed/runner'
11
+ require 'glazed/steps'
12
+ require 'glazed/workflow'
13
+
14
+ module Glazed
15
+ class Coordinator
16
+ # Performs {Proto::V1::CoordinatorTask}s for an external coordinator, such
17
+ # as AWS Step Functions, which runs each step of a workflow as its own
18
+ # call and passes the protobuf results between them. Node failures are
19
+ # returned as {Proto::V1::StepFailure}s rather than raised, so the
20
+ # coordinator carries them to the end of the batch.
21
+ #
22
+ # @example
23
+ # runner = Glazed::Coordinator::TaskRunner.new(coordinator:, workflows:, inventory: -> { Glazed::Sources.nodes(config.inventory.to_a) })
24
+ # batches = runner.perform(Glazed::Proto::V1::CoordinatorTask.new(prepare: {}))
25
+ class TaskRunner
26
+ # Cycles a coordinator execution runs when the task sets none.
27
+ DEFAULT_MAX_CYCLES = 20 #: Integer
28
+
29
+ #: type result = Proto::V1::Armed | Proto::V1::PreparedBatches | Proto::V1::NodeOutcome | Proto::V1::Batch
30
+
31
+ # @param coordinator [Coordinator] changes the node schedules
32
+ # @param workflows [Hash{String => Workflow}] the orchestrated workflows, by name
33
+ # @param inventory [Proc] loads the current nodes
34
+ # @param listener [Proc] receives events of the tasks and steps
35
+ # @raise [ArgumentError] when the workflows differ from those the coordinator orchestrates
36
+ #: (coordinator: Coordinator, workflows: Hash[String, Workflow], inventory: Orchestrator::inventory, ?listener: Runner::listener) -> void
37
+ def initialize(coordinator:, workflows:, inventory:, listener: ->(_event) {})
38
+ unless workflows.keys.sort == coordinator.workflows.sort
39
+ raise ArgumentError, "workflows #{workflows.keys.sort} differ from the orchestrated #{coordinator.workflows.sort}"
40
+ end
41
+
42
+ @coordinator = coordinator #: Coordinator
43
+ @workflows = workflows #: Hash[String, Workflow]
44
+ @inventory = inventory #: Orchestrator::inventory
45
+ @listener = listener #: Runner::listener
46
+ end
47
+
48
+ # @param task [Proto::V1::CoordinatorTask] the task
49
+ # @return [Proto::V1::Armed, Proto::V1::PreparedBatches, Proto::V1::NodeOutcome, Proto::V1::Batch] its result
50
+ # @raise [InvalidMessage] when the task breaks the rules of its schema
51
+ # @raise [ArgumentError] when the task names an unknown workflow or step
52
+ # @raise [Glazed::Error] when the inventory or the state backend fails
53
+ #: (Proto::V1::CoordinatorTask task) -> TaskRunner::result
54
+ def perform(task)
55
+ Validation.validate!(task)
56
+ if (check = task.check) then check(check)
57
+ elsif task.prepare then prepare
58
+ elsif (node_step = task.node_step) then node_step(node_step)
59
+ elsif (batch_step = task.batch_step) then batch_step(batch_step)
60
+ else complete(Must.because(task.complete) { 'validation requires a task' })
61
+ end
62
+ end
63
+
64
+ # @param task [Proto::V1::CheckTask] the task
65
+ # @return [Proto::V1::Armed] when to wake up, if at all
66
+ #: (Proto::V1::CheckTask task) -> Proto::V1::Armed
67
+ private def check(task)
68
+ nodes = load_inventory.keys
69
+ @coordinator.sync_inventory(nodes)
70
+ @coordinator.signal(task.signals.to_a, nodes:) unless task.signals.empty?
71
+ wake = @coordinator.arm(
72
+ token: task.token, cycle: task.cycle, max_cycles: task.max_cycles.zero? ? DEFAULT_MAX_CYCLES : task.max_cycles,
73
+ grace: task.grace_seconds.zero? ? nil : task.grace_seconds,
74
+ )
75
+ Proto::V1::Armed.new(wait_until: wake && Google::Protobuf::Timestamp.from_time(wake))
76
+ end
77
+
78
+ # @return [Proto::V1::PreparedBatches] a batch of the due nodes of each workflow
79
+ #: () -> Proto::V1::PreparedBatches
80
+ private def prepare
81
+ inventory = load_inventory
82
+ @coordinator.sync_inventory(inventory.keys)
83
+ batches = @workflows.keys.to_h do |workflow|
84
+ owner, names = @coordinator.claim(workflow)
85
+ nodes = names.filter_map { inventory[it] }.map { Proto::V1::NodeOutcome.new(state: Proto::V1::NodeState.new(node: it)) }
86
+ notify(level: 'info', workflow:, message: 'batch started', nodes: nodes.map { it.state&.node&.name.to_s }) unless nodes.empty?
87
+ [workflow, Proto::V1::Batch.new(workflow:, owner:, nodes:)]
88
+ end
89
+ Proto::V1::PreparedBatches.new(batches:)
90
+ end
91
+
92
+ # @param task [Proto::V1::NodeStepTask] the task
93
+ # @return [Proto::V1::NodeOutcome] the node's outcome after the step
94
+ # @raise [ArgumentError] when the step is unknown or not a per-node step
95
+ #: (Proto::V1::NodeStepTask task) -> Proto::V1::NodeOutcome
96
+ private def node_step(task)
97
+ step = step(task.workflow, task.step, Steps::NodeStep)
98
+ outcome = Must.because(task.outcome) { 'validation requires outcome' }
99
+ return outcome if outcome.failure
100
+
101
+ state = Must.because(outcome.state) { 'validation requires state' }
102
+ node = state.node&.name.to_s
103
+ begin
104
+ after = step.call(state)
105
+ rescue StandardError => e
106
+ # One unreachable or misbehaving device must not stop the others.
107
+ notify(level: 'error', step: step.name, node:, error: e.class.name, message: e.message)
108
+ return Proto::V1::NodeOutcome.new(state:, failure: Proto::V1::StepFailure.new(step: step.name, error: e.message))
109
+ end
110
+ notify(level: 'info', step: step.name, node:, changed: changed?(after, step), message: 'done')
111
+ Proto::V1::NodeOutcome.new(state: after)
112
+ end
113
+
114
+ # @param task [Proto::V1::BatchStepTask] the task
115
+ # @return [Proto::V1::Batch] the batch after the step
116
+ # @raise [ArgumentError] when the step is unknown or not a batch step
117
+ #: (Proto::V1::BatchStepTask task) -> Proto::V1::Batch
118
+ private def batch_step(task)
119
+ batch = Must.because(task.batch) { 'validation requires batch' }
120
+ step = step(batch.workflow, task.step, Steps::BatchStep)
121
+ return batch unless batch.error.empty?
122
+
123
+ passing = batch.nodes.filter_map { it.state unless it.failure }
124
+ begin
125
+ after = step.call(Proto::V1::BatchState.new(nodes: passing)).nodes.to_h { [it.node&.name, it] }
126
+ rescue StandardError => e
127
+ notify(level: 'error', workflow: batch.workflow, step: step.name, error: e.class.name, message: e.message)
128
+ return batch.dup.tap { it.error = "#{step.name}: #{e.message}" }
129
+ end
130
+ notify(level: 'info', workflow: batch.workflow, step: step.name, nodes: after.size, message: 'done')
131
+ nodes = batch.nodes.map do |outcome|
132
+ state = after[outcome.state&.node&.name]
133
+ outcome.failure || state.nil? ? outcome : Proto::V1::NodeOutcome.new(state:)
134
+ end
135
+ Proto::V1::Batch.new(workflow: batch.workflow, owner: batch.owner, nodes:)
136
+ end
137
+
138
+ # @param task [Proto::V1::CompleteTask] the task
139
+ # @return [Proto::V1::Batch] the batch, unchanged
140
+ #: (Proto::V1::CompleteTask task) -> Proto::V1::Batch
141
+ private def complete(task)
142
+ batch = Must.because(task.batch) { 'validation requires batch' }
143
+ names = batch.nodes.map { it.state&.node&.name.to_s }
144
+ errors = {} #: Hash[String, String]
145
+ batch.nodes.each do |outcome|
146
+ failure = outcome.failure
147
+ error = failure ? "#{failure.step}: #{failure.error}" : batch.error
148
+ errors[outcome.state&.node&.name.to_s] = error unless error.empty?
149
+ end
150
+ @coordinator.complete(batch.workflow, owner: batch.owner, names:, errors:)
151
+ notify(level: 'info', workflow: batch.workflow, message: 'batch finished', nodes: names.size, failed: errors.size)
152
+ batch
153
+ end
154
+
155
+ # @return [Hash{String => Proto::V1::Node}] every node in the inventory, by name
156
+ #: () -> Hash[String, Proto::V1::Node]
157
+ private def load_inventory
158
+ nodes = @inventory.call.to_h { [it.name, it] }
159
+ notify(level: 'info', message: 'inventory loaded', nodes: nodes.size)
160
+ nodes
161
+ end
162
+
163
+ # @param workflow [String] workflow name
164
+ # @param name [String] step name
165
+ # @param kind [Class] {Steps::NodeStep} or {Steps::BatchStep}
166
+ # @return [Steps::Base] the step
167
+ # @raise [ArgumentError] when the step is unknown or of another kind
168
+ #: [S < Steps::Base] (String workflow, String name, Class[S] kind) -> S
169
+ private def step(workflow, name, kind)
170
+ steps = @workflows.fetch(workflow) { raise ArgumentError, "workflow #{workflow.inspect} is not orchestrated" }.steps
171
+ found = steps.find { it.name == name } or raise ArgumentError, "workflow #{workflow.inspect} has no step #{name.inspect}"
172
+ raise ArgumentError, "#{workflow}: #{name} is not a #{kind.name}" unless found.is_a?(kind)
173
+
174
+ found
175
+ end
176
+
177
+ # @param state [Proto::V1::NodeState] state after the step
178
+ # @param step [Steps::Base] the step
179
+ # @return [Boolean] whether any artifact the step produced changed stored content
180
+ #: (Proto::V1::NodeState state, Steps::Base step) -> bool
181
+ private def changed?(state, step) = step.outputs.any? { state.artifacts[it]&.changed }
182
+
183
+ # @param event [Hash{Symbol => String, Integer, Boolean, Array<String>, nil}] event fields
184
+ # @return [void]
185
+ #: (**Runner::event_value event) -> void
186
+ private def notify(**event) = @listener.call(event)
187
+ end
188
+ end
189
+ end