floe 0.7.1 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +22 -3
- data/README.md +13 -0
- data/exe/floe +50 -21
- data/floe.gemspec +2 -1
- data/lib/floe/version.rb +1 -1
- data/lib/floe/workflow/context.rb +4 -0
- data/lib/floe/workflow/runner/docker.rb +103 -4
- data/lib/floe/workflow/runner/kubernetes.rb +86 -1
- data/lib/floe/workflow/runner/podman.rb +29 -1
- data/lib/floe/workflow/runner.rb +27 -10
- data/lib/floe/workflow/state.rb +10 -10
- data/lib/floe/workflow/states/non_terminal_mixin.rb +5 -0
- data/lib/floe/workflow/states/task.rb +4 -3
- data/lib/floe/workflow/states/wait.rb +2 -3
- data/lib/floe/workflow.rb +75 -22
- data/lib/floe.rb +20 -0
- metadata +18 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 8c7a74a5297258d481fb588ae0fa6eb1b22b7ecf5c049865b77ad23d6fb135cb
|
4
|
+
data.tar.gz: 82f73726e293e5345d3e7fa55a0049f881f2dce6e6d46570d2352968907c04b9
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 32d58e28cd76d936f31f9af2c1091d8a7dd930e47a2197b532c02d6d48df2c82feee696072af701aeca5f2af5437040ea3bace5df622c1ad5d0e47e388884ad2
|
7
|
+
data.tar.gz: 1ee0628fbfde496d00fae67812ac869582b1aca754aca7cf44caa7170edc1b0f6c9100a587770d6f6bb97bc0a948dd8fe0123fd108b9ed037ad71bc62bb8e104
|
data/CHANGELOG.md
CHANGED
@@ -4,8 +4,26 @@ This project adheres to [Semantic Versioning](http://semver.org/).
|
|
4
4
|
|
5
5
|
## [Unreleased]
|
6
6
|
|
7
|
-
## [0.
|
7
|
+
## [0.9.0] - 2024-02-19
|
8
|
+
### Changed
|
9
|
+
- Default to wait indefinitely ([#157](https://github.com/ManageIQ/floe/pull/157))
|
10
|
+
- Create docker runners factory and add scheme ([#152](https://github.com/ManageIQ/floe/pull/152))
|
11
|
+
- Add a watch method to Workflow::Runner for event driven updates ([#95](https://github.com/ManageIQ/floe/pull/95))
|
12
|
+
|
13
|
+
### Fixed
|
14
|
+
- Fix waiting on extremely short durations ([#160](https://github.com/ManageIQ/floe/pull/160))
|
15
|
+
- Fix wait state missing finish ([#159](https://github.com/ManageIQ/floe/pull/159))
|
16
|
+
|
17
|
+
## [0.8.0] - 2024-01-17
|
18
|
+
### Added
|
19
|
+
- Add CLI shorthand options for docker runner ([#147](https://github.com/ManageIQ/floe/pull/147))
|
20
|
+
- Run multiple workflows in exe/floe ([#149](https://github.com/ManageIQ/floe/pull/149))
|
21
|
+
- Add secure options for passing credentials via command-line ([#151](https://github.com/ManageIQ/floe/pull/151))
|
22
|
+
- Add a Docker Runner pull-policy option ([#155](https://github.com/ManageIQ/floe/pull/155))
|
23
|
+
|
8
24
|
### Fixed
|
25
|
+
- Fix podman with empty output ([#150](https://github.com/ManageIQ/floe/pull/150))
|
26
|
+
- Fix run_container logger saying docker when using podman ([#154](https://github.com/ManageIQ/floe/pull/154))
|
9
27
|
- Ensure that workflow credentials is not-nil ([#156](https://github.com/ManageIQ/floe/pull/156))
|
10
28
|
|
11
29
|
## [0.7.0] - 2023-12-18
|
@@ -118,8 +136,9 @@ This project adheres to [Semantic Versioning](http://semver.org/).
|
|
118
136
|
### Added
|
119
137
|
- Initial release
|
120
138
|
|
121
|
-
[Unreleased]: https://github.com/ManageIQ/floe/compare/v0.
|
122
|
-
[0.
|
139
|
+
[Unreleased]: https://github.com/ManageIQ/floe/compare/v0.9.0...HEAD
|
140
|
+
[0.9.0]: https://github.com/ManageIQ/floe/compare/v0.8.0...v0.9.0
|
141
|
+
[0.8.0]: https://github.com/ManageIQ/floe/compare/v0.7.0...v0.8.0
|
123
142
|
[0.7.0]: https://github.com/ManageIQ/floe/compare/v0.6.1...v0.7.0
|
124
143
|
[0.6.1]: https://github.com/ManageIQ/floe/compare/v0.6.0...v0.6.1
|
125
144
|
[0.6.0]: https://github.com/ManageIQ/floe/compare/v0.5.0...v0.6.0
|
data/README.md
CHANGED
@@ -51,6 +51,16 @@ You can provide that at runtime via the `--credentials` parameter:
|
|
51
51
|
bundle exec ruby exe/floe --workflow my-workflow.asl --credentials='{"roleArn": "arn:aws:iam::111122223333:role/LambdaRole"}'
|
52
52
|
```
|
53
53
|
|
54
|
+
Or if you are running the floe command programmatically you can securely provide the credentials via a stdin pipe via `--credentials=-`:
|
55
|
+
```
|
56
|
+
echo '{"roleArn": "arn:aws:iam::111122223333:role/LambdaRole"}' | bundle exec ruby exe/floe --workflow my-workflow.asl --credentials -
|
57
|
+
```
|
58
|
+
|
59
|
+
Or you can pass a file path with the `--credentials-file` parameter:
|
60
|
+
```
|
61
|
+
bundle exec ruby exe/floe --workflow my-workflow.asl --credentials-file /tmp/20231218-80537-kj494t
|
62
|
+
```
|
63
|
+
|
54
64
|
If you need to set a credential at runtime you can do that by using the `"ResultPath": "$.Credentials"` directive, for example to user a username/password to login and get a Bearer token:
|
55
65
|
|
56
66
|
```
|
@@ -152,6 +162,7 @@ end
|
|
152
162
|
Options supported by the Docker docker runner are:
|
153
163
|
|
154
164
|
* `network` - What docker to connect the container to, defaults to `"bridge"`. If you need access to host resources for development you can pass `network=host`.
|
165
|
+
* `pull-policy` - Pull image policy. The default is missing. Allowed values: always, missing, never
|
155
166
|
|
156
167
|
#### Podman
|
157
168
|
|
@@ -161,6 +172,7 @@ Options supported by the podman docker runner are:
|
|
161
172
|
* `log-level=string` - Log messages above specified level (trace, debug, info, warn, warning, error, fatal, panic)
|
162
173
|
* `network=string` - What docker to connect the container to, defaults to `"bridge"`. If you need access to host resources for development you can pass `network=host`.
|
163
174
|
* `noout=boolean` - do not output to stdout
|
175
|
+
* `pull-policy=string` - Pull image policy. The default is missing. Allowed values: always, missing, never, newer
|
164
176
|
* `root=string` - Path to the root directory in which data, including images, is stored
|
165
177
|
* `runroot=string` - Path to the 'run directory' where all state information is stored
|
166
178
|
* `runtime=string` - Path to the OCI-compatible binary used to run containers
|
@@ -179,6 +191,7 @@ Options supported by the kubernetes docker runner are:
|
|
179
191
|
* `kubeconfig` - Path to a kubeconfig file, defaults to `KUBECONFIG` environment variable or `~/.kube/config`
|
180
192
|
* `kubeconfig_context` - Context to use in the kubeconfig file, defaults to `"default"`
|
181
193
|
* `namespace` - Namespace to use when creating kubernetes resources, defaults to `"default"`
|
194
|
+
* `pull-policy` - Pull image policy. The default is Always. Allowed values: IfNotPresent, Always, Never
|
182
195
|
* `server` - A kubernetes API Server URL, overrides anything in your kubeconfig file. If set `KUBERNETES_SERVICE_HOST` and `KUBERNETES_SERVICE_PORT` will be used
|
183
196
|
* `token` - A bearer_token to use to authenticate to the kubernetes API, overrides anything in your kubeconfig file. If present, `/run/secrets/kubernetes.io/serviceaccount/token` will be used
|
184
197
|
* `ca_file` - Path to a certificate-authority file for the kubernetes API, only valid if server and token are passed. If present `/run/secrets/kubernetes.io/serviceaccount/ca.crt` will be used
|
data/exe/floe
CHANGED
@@ -6,35 +6,64 @@ require "optimist"
|
|
6
6
|
|
7
7
|
opts = Optimist.options do
|
8
8
|
version("v#{Floe::VERSION}\n")
|
9
|
-
|
10
|
-
|
11
|
-
opt :
|
12
|
-
opt :
|
13
|
-
opt :
|
9
|
+
usage("[options] workflow input [workflow2 input2]")
|
10
|
+
|
11
|
+
opt :workflow, "Path to your workflow json (legacy)", :type => :string
|
12
|
+
opt :input, "JSON payload to input to the workflow (legacy)", :type => :string
|
13
|
+
opt :credentials, "JSON payload with credentials", :type => :string
|
14
|
+
opt :credentials_file, "Path to a file with credentials", :type => :string
|
15
|
+
opt :docker_runner, "Type of runner for docker images", :type => :string, :short => 'r'
|
16
|
+
opt :docker_runner_options, "Options to pass to the runner", :type => :strings, :short => 'o'
|
17
|
+
|
18
|
+
opt :docker, "Use docker to run images (short for --docker_runner=docker)", :type => :boolean
|
19
|
+
opt :podman, "Use podman to run images (short for --docker_runner=podman)", :type => :boolean
|
20
|
+
opt :kubernetes, "Use kubernetes to run images (short for --docker_runner=kubernetes)", :type => :boolean
|
14
21
|
end
|
15
22
|
|
16
|
-
|
23
|
+
# legacy support for --workflow
|
24
|
+
args = ARGV.empty? ? [opts[:workflow], opts[:input]] : ARGV
|
25
|
+
Optimist.die(:workflow, "must be specified") if args.empty?
|
26
|
+
|
27
|
+
# shortcut support
|
28
|
+
opts[:docker_runner] ||= "docker" if opts[:docker]
|
29
|
+
opts[:docker_runner] ||= "podman" if opts[:podman]
|
30
|
+
opts[:docker_runner] ||= "kubernetes" if opts[:kubernetes]
|
17
31
|
|
18
32
|
require "logger"
|
19
33
|
Floe.logger = Logger.new($stdout)
|
20
34
|
|
21
|
-
runner_klass = case opts[:docker_runner]
|
22
|
-
when "docker"
|
23
|
-
Floe::Workflow::Runner::Docker
|
24
|
-
when "podman"
|
25
|
-
Floe::Workflow::Runner::Podman
|
26
|
-
when "kubernetes"
|
27
|
-
Floe::Workflow::Runner::Kubernetes
|
28
|
-
end
|
29
|
-
|
30
35
|
runner_options = opts[:docker_runner_options].to_h { |opt| opt.split("=", 2) }
|
31
36
|
|
32
|
-
|
37
|
+
begin
|
38
|
+
Floe.set_runner("docker", opts[:docker_runner], runner_options)
|
39
|
+
rescue ArgumentError => e
|
40
|
+
Optimist.die(:docker_runner, e.message)
|
41
|
+
end
|
42
|
+
|
43
|
+
credentials =
|
44
|
+
if opts[:credentials_given]
|
45
|
+
opts[:credentials] == "-" ? $stdin.read : opts[:credentials]
|
46
|
+
elsif opts[:credentials_file_given]
|
47
|
+
File.read(opts[:credentials_file])
|
48
|
+
end
|
33
49
|
|
34
|
-
|
35
|
-
|
50
|
+
workflows =
|
51
|
+
args.each_slice(2).map do |workflow, input|
|
52
|
+
context = Floe::Workflow::Context.new(:input => input || opts[:input] || "{}")
|
53
|
+
Floe::Workflow.load(workflow, context, credentials)
|
54
|
+
end
|
55
|
+
|
56
|
+
# run
|
57
|
+
|
58
|
+
Floe::Workflow.wait(workflows, &:run_nonblock)
|
59
|
+
|
60
|
+
# display status
|
61
|
+
|
62
|
+
workflows.each do |workflow|
|
63
|
+
puts "", "#{workflow.name}#{" (#{workflow.status})" unless workflow.context.success?}", "===" if workflows.size > 1
|
64
|
+
puts workflow.output.inspect
|
65
|
+
end
|
36
66
|
|
37
|
-
|
67
|
+
# exit status
|
38
68
|
|
39
|
-
|
40
|
-
exit workflow.status == "success" ? 0 : 1
|
69
|
+
exit workflows.all? { |workflow| workflow.context.success? } ? 0 : 1
|
data/floe.gemspec
CHANGED
@@ -29,7 +29,8 @@ Gem::Specification.new do |spec|
|
|
29
29
|
spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
|
30
30
|
spec.require_paths = ["lib"]
|
31
31
|
|
32
|
-
spec.add_dependency "awesome_spawn", "~>1.
|
32
|
+
spec.add_dependency "awesome_spawn", "~>1.6"
|
33
|
+
spec.add_dependency "io-wait"
|
33
34
|
spec.add_dependency "jsonpath", "~>1.1"
|
34
35
|
spec.add_dependency "kubeclient", "~>4.7"
|
35
36
|
spec.add_dependency "optimist", "~>3.0"
|
data/lib/floe/version.rb
CHANGED
@@ -10,11 +10,13 @@ module Floe
|
|
10
10
|
|
11
11
|
def initialize(options = {})
|
12
12
|
require "awesome_spawn"
|
13
|
+
require "io/wait"
|
13
14
|
require "tempfile"
|
14
15
|
|
15
16
|
super
|
16
17
|
|
17
|
-
@network
|
18
|
+
@network = options.fetch("network", "bridge")
|
19
|
+
@pull_policy = options["pull-policy"]
|
18
20
|
end
|
19
21
|
|
20
22
|
def run_async!(resource, env = {}, secrets = {})
|
@@ -44,10 +46,63 @@ module Floe
|
|
44
46
|
delete_secret(secrets_file) if secrets_file
|
45
47
|
end
|
46
48
|
|
49
|
+
def wait(timeout: nil, events: %i[create update delete], &block)
|
50
|
+
until_timestamp = Time.now.utc + timeout if timeout
|
51
|
+
|
52
|
+
r, w = IO.pipe
|
53
|
+
|
54
|
+
pid = AwesomeSpawn.run_detached(
|
55
|
+
self.class::DOCKER_COMMAND, :err => :out, :out => w, :params => wait_params(until_timestamp)
|
56
|
+
)
|
57
|
+
|
58
|
+
w.close
|
59
|
+
|
60
|
+
loop do
|
61
|
+
readable_timeout = until_timestamp - Time.now.utc if until_timestamp
|
62
|
+
|
63
|
+
# Wait for our end of the pipe to be readable and if it didn't timeout
|
64
|
+
# get the events from stdout
|
65
|
+
next if r.wait_readable(readable_timeout).nil?
|
66
|
+
|
67
|
+
# Get all events while the pipe is readable
|
68
|
+
notices = []
|
69
|
+
while r.ready?
|
70
|
+
notice = r.gets
|
71
|
+
|
72
|
+
# If the process has exited `r.gets` returns `nil` and the pipe is
|
73
|
+
# always `ready?`
|
74
|
+
break if notice.nil?
|
75
|
+
|
76
|
+
event, runner_context = parse_notice(notice)
|
77
|
+
next if event.nil? || !events.include?(event)
|
78
|
+
|
79
|
+
notices << [event, runner_context]
|
80
|
+
end
|
81
|
+
|
82
|
+
# If we're given a block yield the events otherwise return them
|
83
|
+
if block
|
84
|
+
notices.each(&block)
|
85
|
+
else
|
86
|
+
# Terminate the `docker events` process before returning the events
|
87
|
+
sigterm(pid)
|
88
|
+
|
89
|
+
return notices
|
90
|
+
end
|
91
|
+
|
92
|
+
# Check that the `docker events` process is still alive
|
93
|
+
Process.kill(0, pid)
|
94
|
+
rescue Errno::ESRCH
|
95
|
+
# Break out of the loop if the `docker events` process has exited
|
96
|
+
break
|
97
|
+
end
|
98
|
+
ensure
|
99
|
+
r.close
|
100
|
+
end
|
101
|
+
|
47
102
|
def status!(runner_context)
|
48
103
|
return if runner_context.key?("Error")
|
49
104
|
|
50
|
-
runner_context["container_state"] = inspect_container(runner_context["container_ref"])
|
105
|
+
runner_context["container_state"] = inspect_container(runner_context["container_ref"])&.dig("State")
|
51
106
|
end
|
52
107
|
|
53
108
|
def running?(runner_context)
|
@@ -72,7 +127,7 @@ module Floe
|
|
72
127
|
def run_container(image, env, secrets_file)
|
73
128
|
params = run_container_params(image, env, secrets_file)
|
74
129
|
|
75
|
-
logger.debug("Running #{AwesomeSpawn.build_command_line(
|
130
|
+
logger.debug("Running #{AwesomeSpawn.build_command_line(self.class::DOCKER_COMMAND, params)}")
|
76
131
|
|
77
132
|
result = docker!(*params)
|
78
133
|
result.output
|
@@ -83,14 +138,52 @@ module Floe
|
|
83
138
|
params << :detach
|
84
139
|
params += env.map { |k, v| [:e, "#{k}=#{v}"] }
|
85
140
|
params << [:e, "_CREDENTIALS=/run/secrets"] if secrets_file
|
141
|
+
params << [:pull, @pull_policy] if @pull_policy
|
86
142
|
params << [:net, "host"] if @network == "host"
|
87
143
|
params << [:v, "#{secrets_file}:/run/secrets:z"] if secrets_file
|
88
144
|
params << [:name, container_name(image)]
|
89
145
|
params << image
|
90
146
|
end
|
91
147
|
|
148
|
+
def wait_params(until_timestamp)
|
149
|
+
params = ["events", [:format, "{{json .}}"], [:filter, "type=container"], [:since, Time.now.utc.to_i]]
|
150
|
+
params << [:until, until_timestamp.to_i] if until_timestamp
|
151
|
+
params
|
152
|
+
end
|
153
|
+
|
154
|
+
def parse_notice(notice)
|
155
|
+
notice = JSON.parse(notice)
|
156
|
+
|
157
|
+
status = notice["status"]
|
158
|
+
event = docker_event_status_to_event(status)
|
159
|
+
running = event != :delete
|
160
|
+
|
161
|
+
name, exit_code = notice.dig("Actor", "Attributes")&.values_at("name", "exitCode")
|
162
|
+
|
163
|
+
runner_context = {"container_ref" => name, "container_state" => {"Running" => running, "ExitCode" => exit_code.to_i}}
|
164
|
+
|
165
|
+
[event, runner_context]
|
166
|
+
rescue JSON::ParserError
|
167
|
+
[]
|
168
|
+
end
|
169
|
+
|
170
|
+
def docker_event_status_to_event(status)
|
171
|
+
case status
|
172
|
+
when "create"
|
173
|
+
:create
|
174
|
+
when "start"
|
175
|
+
:update
|
176
|
+
when "die", "destroy"
|
177
|
+
:delete
|
178
|
+
else
|
179
|
+
:unkonwn
|
180
|
+
end
|
181
|
+
end
|
182
|
+
|
92
183
|
def inspect_container(container_id)
|
93
|
-
JSON.parse(docker!("inspect", container_id).output)
|
184
|
+
JSON.parse(docker!("inspect", container_id).output).first
|
185
|
+
rescue
|
186
|
+
nil
|
94
187
|
end
|
95
188
|
|
96
189
|
def delete_container(container_id)
|
@@ -114,6 +207,12 @@ module Floe
|
|
114
207
|
secrets_file.path
|
115
208
|
end
|
116
209
|
|
210
|
+
def sigterm(pid)
|
211
|
+
Process.kill("TERM", pid)
|
212
|
+
rescue Errno::ESRCH
|
213
|
+
nil
|
214
|
+
end
|
215
|
+
|
117
216
|
def global_docker_options
|
118
217
|
[]
|
119
218
|
end
|
@@ -40,6 +40,7 @@ module Floe
|
|
40
40
|
|
41
41
|
@namespace = options.fetch("namespace", "default")
|
42
42
|
|
43
|
+
@pull_policy = options["pull-policy"]
|
43
44
|
@task_service_account = options["task_service_account"]
|
44
45
|
|
45
46
|
super
|
@@ -52,7 +53,7 @@ module Floe
|
|
52
53
|
name = container_name(image)
|
53
54
|
secret = create_secret!(secrets) if secrets && !secrets.empty?
|
54
55
|
|
55
|
-
runner_context = {"container_ref" => name, "secrets_ref" => secret}
|
56
|
+
runner_context = {"container_ref" => name, "container_state" => {"phase" => "Pending"}, "secrets_ref" => secret}
|
56
57
|
|
57
58
|
begin
|
58
59
|
create_pod!(name, image, env, secret)
|
@@ -101,6 +102,54 @@ module Floe
|
|
101
102
|
delete_secret(secret) if secret
|
102
103
|
end
|
103
104
|
|
105
|
+
def wait(timeout: nil, events: %i[create update delete])
|
106
|
+
retry_connection = true
|
107
|
+
|
108
|
+
begin
|
109
|
+
watcher = kubeclient.watch_pods(:namespace => namespace)
|
110
|
+
|
111
|
+
retry_connection = true
|
112
|
+
|
113
|
+
if timeout.to_i > 0
|
114
|
+
timeout_thread = Thread.new do
|
115
|
+
sleep(timeout)
|
116
|
+
watcher.finish
|
117
|
+
end
|
118
|
+
end
|
119
|
+
|
120
|
+
watcher.each do |notice|
|
121
|
+
break if error_notice?(notice)
|
122
|
+
|
123
|
+
event = kube_notice_type_to_event(notice.type)
|
124
|
+
next unless events.include?(event)
|
125
|
+
|
126
|
+
runner_context = parse_notice(notice)
|
127
|
+
next if runner_context.nil?
|
128
|
+
|
129
|
+
if block_given?
|
130
|
+
yield [event, runner_context]
|
131
|
+
else
|
132
|
+
timeout_thread&.kill # If we break out before the timeout, kill the timeout thread
|
133
|
+
return [[event, runner_context]]
|
134
|
+
end
|
135
|
+
end
|
136
|
+
rescue Kubeclient::HttpError => err
|
137
|
+
raise unless err.error_code == 401 && retry_connection
|
138
|
+
|
139
|
+
@kubeclient = nil
|
140
|
+
retry_connection = false
|
141
|
+
retry
|
142
|
+
ensure
|
143
|
+
begin
|
144
|
+
watch&.finish
|
145
|
+
rescue
|
146
|
+
nil
|
147
|
+
end
|
148
|
+
|
149
|
+
timeout_thread&.join(0)
|
150
|
+
end
|
151
|
+
end
|
152
|
+
|
104
153
|
private
|
105
154
|
|
106
155
|
attr_reader :ca_file, :kubeconfig_file, :kubeconfig_context, :namespace, :server, :token, :verify_ssl
|
@@ -143,6 +192,7 @@ module Floe
|
|
143
192
|
}
|
144
193
|
}
|
145
194
|
|
195
|
+
spec[:spec][:imagePullPolicy] = @pull_policy if @pull_policy
|
146
196
|
spec[:spec][:serviceAccountName] = @task_service_account if @task_service_account
|
147
197
|
|
148
198
|
if secret
|
@@ -215,6 +265,41 @@ module Floe
|
|
215
265
|
nil
|
216
266
|
end
|
217
267
|
|
268
|
+
def kube_notice_type_to_event(type)
|
269
|
+
case type
|
270
|
+
when "ADDED"
|
271
|
+
:create
|
272
|
+
when "MODIFIED"
|
273
|
+
:update
|
274
|
+
when "DELETED"
|
275
|
+
:delete
|
276
|
+
else
|
277
|
+
:unknown
|
278
|
+
end
|
279
|
+
end
|
280
|
+
|
281
|
+
def error_notice?(notice)
|
282
|
+
return false unless notice.type == "ERROR"
|
283
|
+
|
284
|
+
message = notice.object&.message
|
285
|
+
code = notice.object&.code
|
286
|
+
reason = notice.object&.reason
|
287
|
+
|
288
|
+
logger.warn("Received [#{code} #{reason}], [#{message}]")
|
289
|
+
|
290
|
+
true
|
291
|
+
end
|
292
|
+
|
293
|
+
def parse_notice(notice)
|
294
|
+
return if notice.object.nil?
|
295
|
+
|
296
|
+
pod = notice.object
|
297
|
+
container_ref = pod.metadata.name
|
298
|
+
container_state = pod.to_h[:status].deep_stringify_keys
|
299
|
+
|
300
|
+
{"container_ref" => container_ref, "container_state" => container_state}
|
301
|
+
end
|
302
|
+
|
218
303
|
def kubeclient
|
219
304
|
return @kubeclient unless @kubeclient.nil?
|
220
305
|
|
@@ -16,6 +16,7 @@ module Floe
|
|
16
16
|
@log_level = options["log-level"]
|
17
17
|
@network = options["network"]
|
18
18
|
@noout = options["noout"].to_s == "true" if options.key?("noout")
|
19
|
+
@pull_policy = options["pull-policy"]
|
19
20
|
@root = options["root"]
|
20
21
|
@runroot = options["runroot"]
|
21
22
|
@runtime = options["runtime"]
|
@@ -35,7 +36,8 @@ module Floe
|
|
35
36
|
params << :detach
|
36
37
|
params += env.map { |k, v| [:e, "#{k}=#{v}"] }
|
37
38
|
params << [:e, "_CREDENTIALS=/run/secrets/#{secret}"] if secret
|
38
|
-
params << [:
|
39
|
+
params << [:pull, @pull_policy] if @pull_policy
|
40
|
+
params << [:net, "host"] if @network == "host"
|
39
41
|
params << [:secret, secret] if secret
|
40
42
|
params << [:name, container_name(image)]
|
41
43
|
params << image
|
@@ -53,6 +55,32 @@ module Floe
|
|
53
55
|
nil
|
54
56
|
end
|
55
57
|
|
58
|
+
def parse_notice(notice)
|
59
|
+
id, status, exit_code = JSON.parse(notice).values_at("ID", "Status", "ContainerExitCode")
|
60
|
+
|
61
|
+
event = podman_event_status_to_event(status)
|
62
|
+
running = event != :delete
|
63
|
+
|
64
|
+
runner_context = {"container_ref" => id, "container_state" => {"Running" => running, "ExitCode" => exit_code.to_i}}
|
65
|
+
|
66
|
+
[event, runner_context]
|
67
|
+
rescue JSON::ParserError
|
68
|
+
[]
|
69
|
+
end
|
70
|
+
|
71
|
+
def podman_event_status_to_event(status)
|
72
|
+
case status
|
73
|
+
when "create"
|
74
|
+
:create
|
75
|
+
when "init", "start"
|
76
|
+
:update
|
77
|
+
when "died", "cleanup", "remove"
|
78
|
+
:delete
|
79
|
+
else
|
80
|
+
:unknown
|
81
|
+
end
|
82
|
+
end
|
83
|
+
|
56
84
|
alias podman! docker!
|
57
85
|
|
58
86
|
def global_docker_options
|
data/lib/floe/workflow/runner.rb
CHANGED
@@ -5,29 +5,42 @@ module Floe
|
|
5
5
|
class Runner
|
6
6
|
include Logging
|
7
7
|
|
8
|
-
TYPES = %w[docker podman kubernetes].freeze
|
9
8
|
OUTPUT_MARKER = "__FLOE_OUTPUT__\n"
|
10
9
|
|
11
10
|
def initialize(_options = {})
|
12
11
|
end
|
13
12
|
|
13
|
+
@runners = {}
|
14
14
|
class << self
|
15
|
-
|
15
|
+
# deprecated -- use Floe.set_runner instead
|
16
|
+
def docker_runner=(value)
|
17
|
+
set_runner("docker", value)
|
18
|
+
end
|
16
19
|
|
17
|
-
|
18
|
-
|
20
|
+
# see Floe.set_runner
|
21
|
+
def set_runner(scheme, name_or_instance, options = {})
|
22
|
+
@runners[scheme] =
|
23
|
+
case name_or_instance
|
24
|
+
when "docker", nil
|
25
|
+
Floe::Workflow::Runner::Docker.new(options)
|
26
|
+
when "podman"
|
27
|
+
Floe::Workflow::Runner::Podman.new(options)
|
28
|
+
when "kubernetes"
|
29
|
+
Floe::Workflow::Runner::Kubernetes.new(options)
|
30
|
+
when Floe::Workflow::Runner
|
31
|
+
name_or_instance
|
32
|
+
else
|
33
|
+
raise ArgumentError, "docker runner must be one of: docker, podman, kubernetes"
|
34
|
+
end
|
19
35
|
end
|
20
36
|
|
21
37
|
def for_resource(resource)
|
22
38
|
raise ArgumentError, "resource cannot be nil" if resource.nil?
|
23
39
|
|
40
|
+
# if no runners are set, default docker:// to docker
|
41
|
+
set_runner("docker", "docker") if @runners.empty?
|
24
42
|
scheme = resource.split("://").first
|
25
|
-
|
26
|
-
when "docker"
|
27
|
-
docker_runner
|
28
|
-
else
|
29
|
-
raise "Invalid resource scheme [#{scheme}]"
|
30
|
-
end
|
43
|
+
@runners[scheme] || raise(ArgumentError, "Invalid resource scheme [#{scheme}]")
|
31
44
|
end
|
32
45
|
end
|
33
46
|
|
@@ -55,6 +68,10 @@ module Floe
|
|
55
68
|
def cleanup(_runner_context)
|
56
69
|
raise NotImplementedError, "Must be implemented in a subclass"
|
57
70
|
end
|
71
|
+
|
72
|
+
def wait(timeout: nil, events: %i[create update delete])
|
73
|
+
raise NotImplementedError, "Must be implemented in a subclass"
|
74
|
+
end
|
58
75
|
end
|
59
76
|
end
|
60
77
|
end
|
data/lib/floe/workflow/state.rb
CHANGED
@@ -33,16 +33,12 @@ module Floe
|
|
33
33
|
raise Floe::InvalidWorkflowError, "State name [#{name}] must be less than or equal to 80 characters" if name.length > 80
|
34
34
|
end
|
35
35
|
|
36
|
-
def
|
37
|
-
wait until run_nonblock! == 0
|
38
|
-
end
|
39
|
-
|
40
|
-
def wait(timeout: 5)
|
36
|
+
def wait(timeout: nil)
|
41
37
|
start = Time.now.utc
|
42
38
|
|
43
39
|
loop do
|
44
40
|
return 0 if ready?
|
45
|
-
return Errno::EAGAIN if timeout.zero? || Time.now.utc - start > timeout
|
41
|
+
return Errno::EAGAIN if timeout && (timeout.zero? || Time.now.utc - start > timeout)
|
46
42
|
|
47
43
|
sleep(1)
|
48
44
|
end
|
@@ -97,6 +93,14 @@ module Floe
|
|
97
93
|
context.state.key?("FinishedTime")
|
98
94
|
end
|
99
95
|
|
96
|
+
def waiting?
|
97
|
+
context.state["WaitUntil"] && Time.now.utc <= Time.parse(context.state["WaitUntil"])
|
98
|
+
end
|
99
|
+
|
100
|
+
def wait_until
|
101
|
+
context.state["WaitUntil"] && Time.parse(context.state["WaitUntil"])
|
102
|
+
end
|
103
|
+
|
100
104
|
private
|
101
105
|
|
102
106
|
def wait_until!(seconds: nil, time: nil)
|
@@ -109,10 +113,6 @@ module Floe
|
|
109
113
|
time.iso8601
|
110
114
|
end
|
111
115
|
end
|
112
|
-
|
113
|
-
def waiting?
|
114
|
-
context.state["WaitUntil"] && Time.now.utc <= Time.parse(context.state["WaitUntil"])
|
115
|
-
end
|
116
116
|
end
|
117
117
|
end
|
118
118
|
end
|
@@ -4,6 +4,11 @@ module Floe
|
|
4
4
|
class Workflow
|
5
5
|
module States
|
6
6
|
module NonTerminalMixin
|
7
|
+
def finish
|
8
|
+
context.next_state = end? ? nil : @next
|
9
|
+
super
|
10
|
+
end
|
11
|
+
|
7
12
|
def validate_state_next!
|
8
13
|
raise Floe::InvalidWorkflowError, "Missing \"Next\" field in state [#{name}]" if @next.nil? && !@end
|
9
14
|
raise Floe::InvalidWorkflowError, "\"Next\" [#{@next}] not in \"States\" for state [#{name}]" if @next && !workflow.payload["States"].key?(@next)
|
@@ -46,18 +46,19 @@ module Floe
|
|
46
46
|
end
|
47
47
|
|
48
48
|
def finish
|
49
|
+
super
|
50
|
+
|
49
51
|
output = runner.output(context.state["RunnerContext"])
|
50
52
|
|
51
53
|
if success?
|
52
54
|
output = parse_output(output)
|
53
55
|
context.state["Output"] = process_output(context.input.dup, output)
|
54
|
-
context.next_state = next_state
|
55
56
|
else
|
57
|
+
context.next_state = nil
|
56
58
|
error = parse_error(output)
|
57
59
|
retry_state!(error) || catch_error!(error) || fail_workflow!(error)
|
58
60
|
end
|
59
61
|
|
60
|
-
super
|
61
62
|
ensure
|
62
63
|
runner.cleanup(context.state["RunnerContext"])
|
63
64
|
end
|
@@ -137,8 +138,8 @@ module Floe
|
|
137
138
|
end
|
138
139
|
|
139
140
|
def parse_output(output)
|
140
|
-
return if output.nil?
|
141
141
|
return output if output.kind_of?(Hash)
|
142
|
+
return if output.nil? || output.empty?
|
142
143
|
|
143
144
|
JSON.parse(output.split("\n").last)
|
144
145
|
rescue JSON::ParserError
|
@@ -28,10 +28,9 @@ module Floe
|
|
28
28
|
|
29
29
|
def start(input)
|
30
30
|
super
|
31
|
-
input = input_path.value(context, input)
|
32
31
|
|
33
|
-
|
34
|
-
context.
|
32
|
+
input = input_path.value(context, input)
|
33
|
+
context.output = output_path.value(context, input)
|
35
34
|
|
36
35
|
wait_until!(
|
37
36
|
:seconds => seconds_path ? seconds_path.value(context, input).to_i : seconds,
|
data/lib/floe/workflow.rb
CHANGED
@@ -8,32 +8,86 @@ module Floe
|
|
8
8
|
include Logging
|
9
9
|
|
10
10
|
class << self
|
11
|
-
def load(path_or_io, context = nil, credentials = {})
|
11
|
+
def load(path_or_io, context = nil, credentials = {}, name = nil)
|
12
12
|
payload = path_or_io.respond_to?(:read) ? path_or_io.read : File.read(path_or_io)
|
13
|
-
|
13
|
+
# default the name if it is a filename and none was passed in
|
14
|
+
name ||= path_or_io.respond_to?(:read) ? "stream" : path_or_io.split("/").last.split(".").first
|
15
|
+
|
16
|
+
new(payload, context, credentials, name)
|
14
17
|
end
|
15
18
|
|
16
|
-
def wait(workflows, timeout:
|
19
|
+
def wait(workflows, timeout: nil, &block)
|
20
|
+
workflows = [workflows] if workflows.kind_of?(self)
|
17
21
|
logger.info("checking #{workflows.count} workflows...")
|
18
22
|
|
19
|
-
|
20
|
-
ready
|
23
|
+
run_until = Time.now.utc + timeout if timeout.to_i > 0
|
24
|
+
ready = []
|
25
|
+
queue = Queue.new
|
26
|
+
wait_thread = Thread.new do
|
27
|
+
loop do
|
28
|
+
Runner.for_resource("docker").wait do |event, runner_context|
|
29
|
+
queue.push([event, runner_context])
|
30
|
+
end
|
31
|
+
end
|
32
|
+
end
|
21
33
|
|
22
34
|
loop do
|
23
35
|
ready = workflows.select(&:step_nonblock_ready?)
|
24
|
-
break if
|
25
|
-
|
26
|
-
|
36
|
+
break if block.nil? && !ready.empty?
|
37
|
+
|
38
|
+
ready.each(&block)
|
39
|
+
|
40
|
+
# Break if all workflows are completed or we've exceeded the
|
41
|
+
# requested timeout
|
42
|
+
break if workflows.all?(&:end?)
|
43
|
+
break if timeout && (timeout.zero? || Time.now.utc > run_until)
|
44
|
+
|
45
|
+
# Find the earliest time that we should wakeup if no container events
|
46
|
+
# are caught, either a workflow in a Wait or Retry state or we've
|
47
|
+
# exceeded the requested timeout
|
48
|
+
wait_until = workflows.map(&:wait_until)
|
49
|
+
.unshift(run_until)
|
50
|
+
.compact
|
51
|
+
.min
|
52
|
+
|
53
|
+
# If a workflow is in a waiting state wakeup the main thread when
|
54
|
+
# it will be done sleeping
|
55
|
+
if wait_until
|
56
|
+
sleep_thread = Thread.new do
|
57
|
+
sleep_duration = wait_until - Time.now.utc
|
58
|
+
sleep sleep_duration if sleep_duration > 0
|
59
|
+
queue.push(nil)
|
60
|
+
end
|
61
|
+
end
|
62
|
+
|
63
|
+
loop do
|
64
|
+
# Block until an event is raised
|
65
|
+
event, runner_context = queue.pop
|
66
|
+
break if event.nil?
|
67
|
+
|
68
|
+
# If the event is for one of our workflows set the updated runner_context
|
69
|
+
workflows.each do |workflow|
|
70
|
+
next unless workflow.context.state.dig("RunnerContext", "container_ref") == runner_context["container_ref"]
|
71
|
+
|
72
|
+
workflow.context.state["RunnerContext"] = runner_context
|
73
|
+
end
|
74
|
+
|
75
|
+
break if queue.empty?
|
76
|
+
end
|
77
|
+
ensure
|
78
|
+
sleep_thread&.kill
|
27
79
|
end
|
28
80
|
|
29
81
|
logger.info("checking #{workflows.count} workflows...Complete - #{ready.count} ready")
|
30
82
|
ready
|
83
|
+
ensure
|
84
|
+
wait_thread&.kill
|
31
85
|
end
|
32
86
|
end
|
33
87
|
|
34
|
-
attr_reader :context, :credentials, :payload, :states, :states_by_name, :start_at
|
88
|
+
attr_reader :context, :credentials, :payload, :states, :states_by_name, :start_at, :name
|
35
89
|
|
36
|
-
def initialize(payload, context = nil, credentials = {})
|
90
|
+
def initialize(payload, context = nil, credentials = {}, name = nil)
|
37
91
|
payload = JSON.parse(payload) if payload.kind_of?(String)
|
38
92
|
credentials = JSON.parse(credentials) if credentials.kind_of?(String)
|
39
93
|
context = Context.new(context) unless context.kind_of?(Context)
|
@@ -42,12 +96,13 @@ module Floe
|
|
42
96
|
raise Floe::InvalidWorkflowError, "Missing field \"StartAt\"" if payload["StartAt"].nil?
|
43
97
|
raise Floe::InvalidWorkflowError, "\"StartAt\" not in the \"States\" field" unless payload["States"].key?(payload["StartAt"])
|
44
98
|
|
99
|
+
@name = name
|
45
100
|
@payload = payload
|
46
101
|
@context = context
|
47
102
|
@credentials = credentials || {}
|
48
103
|
@start_at = payload["StartAt"]
|
49
104
|
|
50
|
-
@states = payload["States"].to_a.map { |
|
105
|
+
@states = payload["States"].to_a.map { |state_name, state| State.build!(self, state_name, state) }
|
51
106
|
@states_by_name = @states.each_with_object({}) { |state, result| result[state.name] = state }
|
52
107
|
|
53
108
|
unless context.state.key?("Name")
|
@@ -58,16 +113,6 @@ module Floe
|
|
58
113
|
raise Floe::InvalidWorkflowError, err.message
|
59
114
|
end
|
60
115
|
|
61
|
-
def run!
|
62
|
-
step until end?
|
63
|
-
self
|
64
|
-
end
|
65
|
-
|
66
|
-
def step
|
67
|
-
step_nonblock_wait until step_nonblock == 0
|
68
|
-
self
|
69
|
-
end
|
70
|
-
|
71
116
|
def run_nonblock
|
72
117
|
loop while step_nonblock == 0 && !end?
|
73
118
|
self
|
@@ -80,7 +125,7 @@ module Floe
|
|
80
125
|
current_state.run_nonblock!
|
81
126
|
end
|
82
127
|
|
83
|
-
def step_nonblock_wait(timeout:
|
128
|
+
def step_nonblock_wait(timeout: nil)
|
84
129
|
current_state.wait(:timeout => timeout)
|
85
130
|
end
|
86
131
|
|
@@ -88,6 +133,14 @@ module Floe
|
|
88
133
|
current_state.ready?
|
89
134
|
end
|
90
135
|
|
136
|
+
def waiting?
|
137
|
+
current_state.waiting?
|
138
|
+
end
|
139
|
+
|
140
|
+
def wait_until
|
141
|
+
current_state.wait_until
|
142
|
+
end
|
143
|
+
|
91
144
|
def status
|
92
145
|
context.status
|
93
146
|
end
|
data/lib/floe.rb
CHANGED
@@ -45,7 +45,27 @@ module Floe
|
|
45
45
|
@logger ||= NullLogger.new
|
46
46
|
end
|
47
47
|
|
48
|
+
# Set the logger to use
|
49
|
+
#
|
50
|
+
# @example
|
51
|
+
# require "logger"
|
52
|
+
# Floe.logger = Logger.new($stdout)
|
53
|
+
#
|
54
|
+
# @param logger [Logger] logger to use for logging actions
|
48
55
|
def self.logger=(logger)
|
49
56
|
@logger = logger
|
50
57
|
end
|
58
|
+
|
59
|
+
# Set the runner to use
|
60
|
+
#
|
61
|
+
# @example
|
62
|
+
# Floe.set_runner "docker", kubernetes", {}
|
63
|
+
# Floe.set_runner "docker", Floe::Workflow::Runner::Kubernetes.new({})
|
64
|
+
#
|
65
|
+
# @param scheme [String] scheme Protocol to register (e.g.: docker)
|
66
|
+
# @param name_or_instance [String|Floe::Workflow::Runner] Name of runner to use for docker (e.g.: docker)
|
67
|
+
# @param options [Hash] Options for constructor of the runner (optional)
|
68
|
+
def self.set_runner(scheme, name_or_instance, options = {})
|
69
|
+
Floe::Workflow::Runner.set_runner(scheme, name_or_instance, options)
|
70
|
+
end
|
51
71
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: floe
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.9.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- ManageIQ Developers
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2024-
|
11
|
+
date: 2024-02-19 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: awesome_spawn
|
@@ -16,14 +16,28 @@ dependencies:
|
|
16
16
|
requirements:
|
17
17
|
- - "~>"
|
18
18
|
- !ruby/object:Gem::Version
|
19
|
-
version: '1.
|
19
|
+
version: '1.6'
|
20
20
|
type: :runtime
|
21
21
|
prerelease: false
|
22
22
|
version_requirements: !ruby/object:Gem::Requirement
|
23
23
|
requirements:
|
24
24
|
- - "~>"
|
25
25
|
- !ruby/object:Gem::Version
|
26
|
-
version: '1.
|
26
|
+
version: '1.6'
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: io-wait
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - ">="
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: '0'
|
34
|
+
type: :runtime
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - ">="
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: '0'
|
27
41
|
- !ruby/object:Gem::Dependency
|
28
42
|
name: jsonpath
|
29
43
|
requirement: !ruby/object:Gem::Requirement
|