activejob-temporal 0.3.1 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +34 -0
- data/activejob-temporal.gemspec +2 -0
- data/bin/temporal-worker +12 -24
- data/lib/activejob/temporal/client.rb +34 -0
- data/lib/activejob/temporal/configuration.rb +52 -41
- data/lib/activejob/temporal/credential_refresher.rb +222 -0
- data/lib/activejob/temporal/locales/en.yml +10 -6
- data/lib/activejob/temporal/version.rb +1 -1
- data/lib/activejob/temporal/worker_runtime.rb +1 -1
- metadata +2 -2
- data/lib/activejob/temporal/certificate_watcher.rb +0 -116
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: ebc4386d6eac6522fba437fbdbd136334a3b5c333f6f83ad90fbfbaab5664144
|
|
4
|
+
data.tar.gz: 53cc07919979a66c013f6dd9597064cc73929ee8930a05c351774883760a1b59
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 652237c25d9342062bebd8269328f684c2cfc1d7a630cdbe0c5be9ae0a2f7d6bce0a96f48cd47de8dbad8da7a9fe5dbbf0974cfa7b92a3a646789c4e25c78873
|
|
7
|
+
data.tar.gz: 6503e062254a40daae4d00a97fa66a312e2d57d127c2cecd13218468ec73b9eecc168f8c3f2dcdc7734fb089f4f9b0ea7b53f4369f8e11ee0934ab4e0d05a143
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.5.0] - 2026-08-07
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
- Configuration now fails to validate when `api_key` or `api_key_file` is set while `tls` is left unset. The Temporal SDK enables TLS on its own whenever an API key is present, so leaving `tls` at `nil` silently coupled transport security to an unrelated setting and failed the handshake against a plaintext frontend.
|
|
14
|
+
- The client logs `credential_source_shadowed` when two sources for the same credential are configured (`api_key` alongside `api_key_file`, or `tls` alongside the `tls_*_path` settings). Precedence was already defined but silent.
|
|
15
|
+
|
|
16
|
+
### Changed
|
|
17
|
+
- **Breaking:** `tls_cert_watch` and `api_key_watch` are replaced by a single `credential_watch`, default `true`. A file path is only configured because the credential behind it rotates, so watching every configured credential file is now the default instead of two opt-in flags that could each be forgotten.
|
|
18
|
+
- **Breaking:** `tls_reload_signal` is renamed `reload_signal` (`ACTIVEJOB_TEMPORAL_TLS_RELOAD_SIGNAL` becomes `ACTIVEJOB_TEMPORAL_RELOAD_SIGNAL`). It rebuilds the whole client and re-resolves the API key, not just TLS material.
|
|
19
|
+
- **Breaking:** `CredentialRefresher.from_config` no longer accepts `on_api_key_change`. Applying a rotated token mutates the live connection, so it is the same call in every process and had no caller that overrode it.
|
|
20
|
+
|
|
21
|
+
### Documentation
|
|
22
|
+
- `CredentialRefresher.from_config` and the worker setup guide now state that a process running its own `Temporalio::Worker` must pass `on_tls_change`. The default `reload_client!` closes the client it replaces, which is the one such a worker is still polling on. `bin/temporal-worker` already passes it.
|
|
23
|
+
|
|
24
|
+
### Migration from 0.4.x
|
|
25
|
+
- Rename `tls_cert_watch` / `api_key_watch` to `credential_watch`, or drop them: watching is on by default and applies to whatever credential files are configured.
|
|
26
|
+
- Rename `tls_reload_signal` to `reload_signal`, and `ACTIVEJOB_TEMPORAL_TLS_RELOAD_SIGNAL` to `ACTIVEJOB_TEMPORAL_RELOAD_SIGNAL`.
|
|
27
|
+
- Set `tls` explicitly wherever `api_key` or `api_key_file` is configured: `config.tls = false` for a plaintext in-cluster frontend, `config.tls = true` for Temporal Cloud. Boot now fails until you do.
|
|
28
|
+
- Drop `on_api_key_change:` from any `CredentialRefresher.from_config` call.
|
|
29
|
+
|
|
30
|
+
## [0.4.0] - 2026-08-07
|
|
31
|
+
|
|
32
|
+
### Added
|
|
33
|
+
- `credential_poll_interval` (default 30 seconds) and `credential_file_events` (default false) configure how `tls_cert_watch` and `api_key_watch` detect rotation.
|
|
34
|
+
- `ActiveJob::Temporal::CredentialRefresher` replaces `CertificateWatcher`. Enqueue-side processes can now start it from an initializer instead of scheduling `refresh_api_key!` themselves - see [worker setup](docs/worker_setup.md#api-key-authentication).
|
|
35
|
+
|
|
36
|
+
### Changed
|
|
37
|
+
- Credential rotation is detected by comparing a content digest on an interval rather than by filesystem events. Rotation on mounts whose events never reach the process (or where the `listen` gem is absent) now works, at up to one poll interval of latency. `credential_file_events = true` restores immediate reaction by waking the check early; it never decides whether a reload happens, so a missing `listen` gem costs latency, not correctness.
|
|
38
|
+
- A cert and key pair rotating together now triggers one client rebuild instead of a debounced pair, and TLS plus API key watching share a single watcher instead of two.
|
|
39
|
+
- **Breaking:** `ActiveJob::Temporal::CertificateWatcher` is removed. Applications constructing it directly should use `CredentialRefresher.from_config`.
|
|
40
|
+
|
|
41
|
+
### Fixed
|
|
42
|
+
- Watching a Kubernetes projected or secret volume no longer prints listen's `directory is already being watched!` error at boot. kubelet keeps the live secret in a timestamped sibling of the `..data` symlink, so listen's recursive scan reached the same real directory twice; the versioned copy is now skipped.
|
|
43
|
+
|
|
10
44
|
## [0.3.1] - 2026-08-06
|
|
11
45
|
|
|
12
46
|
### Fixed
|
data/activejob-temporal.gemspec
CHANGED
|
@@ -39,6 +39,8 @@ Gem::Specification.new do |spec|
|
|
|
39
39
|
spec.add_dependency "globalid", ">= 0.3"
|
|
40
40
|
spec.add_dependency "temporalio", ">= 1.4.0", "< 1.7"
|
|
41
41
|
|
|
42
|
+
# listen is an optional runtime dependency, required lazily only when credential_file_events is
|
|
43
|
+
# enabled. Without it credentials still rotate, just on the poll interval instead of instantly.
|
|
42
44
|
spec.add_development_dependency "benchmark-ips", "~> 2.14"
|
|
43
45
|
spec.add_development_dependency "github_changelog_generator", "~> 1.18"
|
|
44
46
|
spec.add_development_dependency "listen", "~> 3.9"
|
data/bin/temporal-worker
CHANGED
|
@@ -107,11 +107,11 @@ if pool_size < 1
|
|
|
107
107
|
exit(1)
|
|
108
108
|
end
|
|
109
109
|
|
|
110
|
-
reload_signal = config.
|
|
110
|
+
reload_signal = config.reload_signal.to_s.sub(/\ASIG/i, "").upcase
|
|
111
111
|
unless Signal.list.key?(reload_signal) && !ActiveJob::Temporal::UNTRAPPABLE_SIGNALS.include?(reload_signal)
|
|
112
112
|
warn(
|
|
113
|
-
"Error:
|
|
114
|
-
"got: #{config.
|
|
113
|
+
"Error: ACTIVEJOB_TEMPORAL_RELOAD_SIGNAL must be a signal name safe to trap, " \
|
|
114
|
+
"got: #{config.reload_signal.inspect}"
|
|
115
115
|
)
|
|
116
116
|
exit(1)
|
|
117
117
|
end
|
|
@@ -260,25 +260,14 @@ reload_signal_thread = Thread.new do
|
|
|
260
260
|
end
|
|
261
261
|
Signal.trap(reload_signal) { reload_signal_queue.push(reload_signal) }
|
|
262
262
|
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
)
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
# Token rotation mutates the live connection (the SDK sends the header per-RPC from the
|
|
273
|
-
# connection's stored options) - no client rebuild, unlike certificate rotation above.
|
|
274
|
-
# Configuration validation guarantees api_key_file is set when api_key_watch is on.
|
|
275
|
-
api_key_watcher = nil
|
|
276
|
-
if config.api_key_watch
|
|
277
|
-
api_key_watcher = ActiveJob::Temporal::CertificateWatcher.new(
|
|
278
|
-
paths: [config.api_key_file],
|
|
279
|
-
reload_callback: -> { ActiveJob::Temporal.refresh_api_key! }
|
|
280
|
-
).start
|
|
281
|
-
end
|
|
263
|
+
# Only the TLS callback is overridden: certificate rotation has to swap the rebuilt client into
|
|
264
|
+
# the running worker, while token rotation mutates the live connection (the SDK sends the header
|
|
265
|
+
# per-RPC from the connection's stored options) and needs no worker awareness.
|
|
266
|
+
credential_refresher = ActiveJob::Temporal::CredentialRefresher.from_config(
|
|
267
|
+
config,
|
|
268
|
+
# Failures are logged by the reloader and retried on the next check, so let them propagate.
|
|
269
|
+
on_tls_change: -> { client_reloader.reload(source: "credential_refresh") }
|
|
270
|
+
).start
|
|
282
271
|
|
|
283
272
|
begin
|
|
284
273
|
worker_health.mark_started!
|
|
@@ -290,8 +279,7 @@ rescue Interrupt
|
|
|
290
279
|
# Temporal worker raises Interrupt on shutdown signals; swallow to allow clean exit.
|
|
291
280
|
ensure
|
|
292
281
|
worker_health.mark_stopped!
|
|
293
|
-
|
|
294
|
-
api_key_watcher&.stop
|
|
282
|
+
credential_refresher.stop
|
|
295
283
|
reload_signal_queue.close
|
|
296
284
|
reload_signal_thread.join(1)
|
|
297
285
|
health_check_server&.stop
|
|
@@ -8,6 +8,7 @@ rescue LoadError
|
|
|
8
8
|
end
|
|
9
9
|
|
|
10
10
|
require_relative "tls_file"
|
|
11
|
+
require_relative "logger"
|
|
11
12
|
|
|
12
13
|
module ActiveJob
|
|
13
14
|
module Temporal
|
|
@@ -121,6 +122,8 @@ module ActiveJob
|
|
|
121
122
|
def connection_kwargs(configuration)
|
|
122
123
|
kwargs = {}
|
|
123
124
|
|
|
125
|
+
warn_on_shadowed_credentials(configuration)
|
|
126
|
+
|
|
124
127
|
tls = tls_options(configuration)
|
|
125
128
|
kwargs[:tls] = tls unless tls.nil?
|
|
126
129
|
|
|
@@ -131,6 +134,37 @@ module ActiveJob
|
|
|
131
134
|
end
|
|
132
135
|
private_class_method :connection_kwargs
|
|
133
136
|
|
|
137
|
+
# Every credential has more than one possible source and the precedence between them is
|
|
138
|
+
# silent, so a config that sets two looks like it works while one of them is ignored.
|
|
139
|
+
# @api private
|
|
140
|
+
def warn_on_shadowed_credentials(configuration)
|
|
141
|
+
if credential_present?(configuration, :api_key) && credential_present?(configuration, :api_key_file)
|
|
142
|
+
Logger.warn("credential_source_shadowed", credential: "api_key", ignored: "api_key_file")
|
|
143
|
+
end
|
|
144
|
+
|
|
145
|
+
return unless configuration.respond_to?(:tls) && !configuration.tls.nil?
|
|
146
|
+
return if tls_path_attributes(configuration).empty?
|
|
147
|
+
|
|
148
|
+
Logger.warn(
|
|
149
|
+
"credential_source_shadowed",
|
|
150
|
+
credential: "tls",
|
|
151
|
+
ignored: tls_path_attributes(configuration).join(", ")
|
|
152
|
+
)
|
|
153
|
+
end
|
|
154
|
+
private_class_method :warn_on_shadowed_credentials
|
|
155
|
+
|
|
156
|
+
def credential_present?(configuration, attribute)
|
|
157
|
+
configuration.respond_to?(attribute) && configuration.public_send(attribute).to_s.strip != ""
|
|
158
|
+
end
|
|
159
|
+
private_class_method :credential_present?
|
|
160
|
+
|
|
161
|
+
def tls_path_attributes(configuration)
|
|
162
|
+
%i[tls_cert_path tls_key_path tls_server_root_ca_cert_path].select do |attribute|
|
|
163
|
+
credential_present?(configuration, attribute)
|
|
164
|
+
end
|
|
165
|
+
end
|
|
166
|
+
private_class_method :tls_path_attributes
|
|
167
|
+
|
|
134
168
|
# Resolves the API key: an explicit api_key wins, otherwise api_key_file is read - at
|
|
135
169
|
# connection build time, and again by {ActiveJob::Temporal.refresh_api_key!} when the
|
|
136
170
|
# token file rotates.
|
|
@@ -122,18 +122,12 @@ module ActiveJob
|
|
|
122
122
|
description: "Optional TLS SNI domain override"
|
|
123
123
|
},
|
|
124
124
|
|
|
125
|
-
|
|
126
|
-
default: false,
|
|
127
|
-
env_var: "ACTIVEJOB_TEMPORAL_TLS_CERT_WATCH",
|
|
128
|
-
type: :boolean,
|
|
129
|
-
description: "Watch TLS certificate files and reload worker clients when they change"
|
|
130
|
-
},
|
|
131
|
-
|
|
132
|
-
tls_reload_signal: {
|
|
125
|
+
reload_signal: {
|
|
133
126
|
default: "HUP",
|
|
134
|
-
env_var: "
|
|
127
|
+
env_var: "ACTIVEJOB_TEMPORAL_RELOAD_SIGNAL",
|
|
135
128
|
type: :string,
|
|
136
|
-
description: "Signal used by workers to
|
|
129
|
+
description: "Signal used by workers to rebuild the Temporal client manually, picking up " \
|
|
130
|
+
"rotated TLS material and a rotated API key"
|
|
137
131
|
},
|
|
138
132
|
|
|
139
133
|
api_key: {
|
|
@@ -151,12 +145,28 @@ module ActiveJob
|
|
|
151
145
|
"projected Kubernetes ServiceAccount token. An explicit api_key takes precedence"
|
|
152
146
|
},
|
|
153
147
|
|
|
154
|
-
|
|
148
|
+
credential_watch: {
|
|
149
|
+
default: true,
|
|
150
|
+
env_var: "ACTIVEJOB_TEMPORAL_CREDENTIAL_WATCH",
|
|
151
|
+
type: :boolean,
|
|
152
|
+
description: "Keep every configured credential file fresh: api_key_file and the TLS " \
|
|
153
|
+
"certificate paths. A file path exists because the credential rotates, so " \
|
|
154
|
+
"this is on by default and watches whatever paths are configured"
|
|
155
|
+
},
|
|
156
|
+
|
|
157
|
+
credential_poll_interval: {
|
|
158
|
+
default: 30,
|
|
159
|
+
env_var: "ACTIVEJOB_TEMPORAL_CREDENTIAL_POLL_INTERVAL",
|
|
160
|
+
type: :integer,
|
|
161
|
+
description: "Seconds between credential file checks when credential_watch is enabled"
|
|
162
|
+
},
|
|
163
|
+
|
|
164
|
+
credential_file_events: {
|
|
155
165
|
default: false,
|
|
156
|
-
env_var: "
|
|
166
|
+
env_var: "ACTIVEJOB_TEMPORAL_CREDENTIAL_FILE_EVENTS",
|
|
157
167
|
type: :boolean,
|
|
158
|
-
description: "
|
|
159
|
-
"
|
|
168
|
+
description: "React to credential file changes immediately instead of waiting for the next " \
|
|
169
|
+
"poll (requires the optional listen gem; the poll interval stays the safety net)"
|
|
160
170
|
},
|
|
161
171
|
|
|
162
172
|
priority_task_queues: {
|
|
@@ -752,6 +762,7 @@ module ActiveJob
|
|
|
752
762
|
validate :validate_payload_storage_settings
|
|
753
763
|
validate :validate_tls_settings
|
|
754
764
|
validate :validate_api_key_settings
|
|
765
|
+
validate :validate_credential_refresh_settings
|
|
755
766
|
validate :validate_worker_registration_settings
|
|
756
767
|
validate :validate_local_activity_helpers
|
|
757
768
|
validate :validate_dependency_wait_settings
|
|
@@ -1038,8 +1049,18 @@ module ActiveJob
|
|
|
1038
1049
|
validate_tls_file_path(:tls_key_path)
|
|
1039
1050
|
validate_tls_file_path(:tls_server_root_ca_cert_path)
|
|
1040
1051
|
validate_tls_domain
|
|
1041
|
-
|
|
1042
|
-
|
|
1052
|
+
validate_tls_decision
|
|
1053
|
+
validate_reload_signal
|
|
1054
|
+
end
|
|
1055
|
+
|
|
1056
|
+
# The Temporal SDK turns TLS on by itself whenever an api_key is present, so leaving `tls`
|
|
1057
|
+
# unset silently couples transport security to an unrelated setting - and fails at connect
|
|
1058
|
+
# against a plaintext in-cluster frontend. Make the caller state the intent instead.
|
|
1059
|
+
def validate_tls_decision
|
|
1060
|
+
return unless tls.nil?
|
|
1061
|
+
return if api_key.to_s.strip.empty? && api_key_file.to_s.strip.empty?
|
|
1062
|
+
|
|
1063
|
+
errors.add(:tls, :requires_explicit_choice)
|
|
1043
1064
|
end
|
|
1044
1065
|
|
|
1045
1066
|
def validate_tls_cert_key_pair
|
|
@@ -1068,28 +1089,22 @@ module ActiveJob
|
|
|
1068
1089
|
errors.add(:tls_domain, :blank)
|
|
1069
1090
|
end
|
|
1070
1091
|
|
|
1071
|
-
def validate_tls_cert_watch
|
|
1072
|
-
unless [true, false].include?(tls_cert_watch)
|
|
1073
|
-
errors.add(:tls_cert_watch, :not_boolean, value: tls_cert_watch.inspect)
|
|
1074
|
-
return
|
|
1075
|
-
end
|
|
1076
|
-
|
|
1077
|
-
return unless tls_cert_watch && tls_watch_paths.empty?
|
|
1078
|
-
|
|
1079
|
-
errors.add(:tls_cert_watch, :requires_paths)
|
|
1080
|
-
end
|
|
1081
|
-
|
|
1082
1092
|
def validate_api_key_settings
|
|
1083
1093
|
validate_tls_file_path(:api_key_file)
|
|
1094
|
+
end
|
|
1084
1095
|
|
|
1085
|
-
|
|
1086
|
-
|
|
1087
|
-
|
|
1096
|
+
def validate_credential_refresh_settings
|
|
1097
|
+
unless [true, false].include?(credential_watch)
|
|
1098
|
+
errors.add(:credential_watch, :not_boolean, value: credential_watch.inspect)
|
|
1099
|
+
end
|
|
1100
|
+
|
|
1101
|
+
unless [true, false].include?(credential_file_events)
|
|
1102
|
+
errors.add(:credential_file_events, :not_boolean, value: credential_file_events.inspect)
|
|
1088
1103
|
end
|
|
1089
1104
|
|
|
1090
|
-
return
|
|
1105
|
+
return if credential_poll_interval.is_a?(Integer) && credential_poll_interval.positive?
|
|
1091
1106
|
|
|
1092
|
-
errors.add(:
|
|
1107
|
+
errors.add(:credential_poll_interval, :invalid, value: credential_poll_interval.inspect)
|
|
1093
1108
|
end
|
|
1094
1109
|
|
|
1095
1110
|
def validate_worker_registration_settings
|
|
@@ -1103,20 +1118,16 @@ module ActiveJob
|
|
|
1103
1118
|
errors.add(:worker_activejob_workloads, :requires_activities)
|
|
1104
1119
|
end
|
|
1105
1120
|
|
|
1106
|
-
def
|
|
1107
|
-
unless
|
|
1108
|
-
errors.add(:
|
|
1121
|
+
def validate_reload_signal
|
|
1122
|
+
unless reload_signal.is_a?(String) && reload_signal.strip.present?
|
|
1123
|
+
errors.add(:reload_signal, :blank)
|
|
1109
1124
|
return
|
|
1110
1125
|
end
|
|
1111
1126
|
|
|
1112
|
-
normalized_signal =
|
|
1127
|
+
normalized_signal = reload_signal.sub(/\ASIG/i, "").upcase
|
|
1113
1128
|
return if Signal.list.key?(normalized_signal) && !UNTRAPPABLE_SIGNALS.include?(normalized_signal)
|
|
1114
1129
|
|
|
1115
|
-
errors.add(:
|
|
1116
|
-
end
|
|
1117
|
-
|
|
1118
|
-
def tls_watch_paths
|
|
1119
|
-
[tls_cert_path, tls_key_path, tls_server_root_ca_cert_path].compact.reject { |path| path.to_s.strip.empty? }
|
|
1130
|
+
errors.add(:reload_signal, :invalid, value: reload_signal.inspect)
|
|
1120
1131
|
end
|
|
1121
1132
|
|
|
1122
1133
|
def callable_accepts_positional_job?(callable)
|
|
@@ -0,0 +1,222 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "digest"
|
|
4
|
+
require_relative "../temporal"
|
|
5
|
+
|
|
6
|
+
module ActiveJob
|
|
7
|
+
module Temporal
|
|
8
|
+
# Keeps credential files fresh in a long-lived process: TLS material, API key files, or
|
|
9
|
+
# anything else read once at boot and rotated underneath the process later.
|
|
10
|
+
#
|
|
11
|
+
# Change detection is a content digest compared on an interval, so the only thing that can fire
|
|
12
|
+
# a callback is the bytes behind a path actually differing. Filesystem events (see :file_events)
|
|
13
|
+
# are an optional accelerator that wakes the loop early; they never decide whether a reload
|
|
14
|
+
# happens, so a missing `listen` gem or a mount without working inotify costs latency, not
|
|
15
|
+
# correctness.
|
|
16
|
+
#
|
|
17
|
+
# @example Worker: rebuild the client on cert rotation, patch the live connection on token rotation
|
|
18
|
+
# refresher = CredentialRefresher.from_config(
|
|
19
|
+
# ActiveJob::Temporal.config,
|
|
20
|
+
# on_tls_change: -> { client_reloader.reload(source: "credential_refresh") }
|
|
21
|
+
# ).start
|
|
22
|
+
#
|
|
23
|
+
# @example Enqueue-side process (web, Sidekiq) with a rotating projected token
|
|
24
|
+
# require "activejob/temporal/credential_refresher"
|
|
25
|
+
# ActiveJob::Temporal::CredentialRefresher.from_config(ActiveJob::Temporal.config).start
|
|
26
|
+
class CredentialRefresher
|
|
27
|
+
# kubelet keeps the live secret in a timestamped sibling of `..data`, so listen's recursive
|
|
28
|
+
# scan reaches the same real directory twice and prints a SymlinkLoop error to stderr.
|
|
29
|
+
# Getting this wrong costs that stderr noise and nothing else: listen only ever nudges the
|
|
30
|
+
# loop, so the digest still decides, and the interval still backs it up.
|
|
31
|
+
#
|
|
32
|
+
# @api private
|
|
33
|
+
KUBERNETES_VERSIONED_DIR = /\A\.\.\d/
|
|
34
|
+
private_constant :KUBERNETES_VERSIONED_DIR
|
|
35
|
+
|
|
36
|
+
DEFAULT_POLL_INTERVAL = 30
|
|
37
|
+
|
|
38
|
+
# Files digested as one unit. Cert and key rotate as separate writes, so grouping them turns
|
|
39
|
+
# a rotation into a single callback instead of a pair.
|
|
40
|
+
Source = Struct.new(:name, :paths, :on_change, keyword_init: true)
|
|
41
|
+
|
|
42
|
+
class << self
|
|
43
|
+
# Builds a refresher for every credential file the configuration names. A path is only
|
|
44
|
+
# configured because the credential behind it rotates, so `credential_watch` gates the
|
|
45
|
+
# whole mechanism rather than each file individually.
|
|
46
|
+
#
|
|
47
|
+
# A token refresh mutates the live connection, so it is the same call in every process.
|
|
48
|
+
# A certificate refresh is not: the default rebuilds the memoized client, which is right
|
|
49
|
+
# for an enqueue-side process but wrong inside a worker.
|
|
50
|
+
#
|
|
51
|
+
# @note A process running a `Temporalio::Worker` MUST pass `on_tls_change`. The worker holds
|
|
52
|
+
# its own client reference, and the default `reload_client!` closes the client it replaces
|
|
53
|
+
# - which is the one the worker is still polling on. Pass a callback that assigns the fresh
|
|
54
|
+
# client to the worker, as {WorkerClientReloader} does.
|
|
55
|
+
#
|
|
56
|
+
# @param configuration [Configuration] the gem configuration
|
|
57
|
+
# @param on_tls_change [#call] invoked when any TLS file changes; see the note above
|
|
58
|
+
# @param logger [#log_event, #warn, #error] structured logger
|
|
59
|
+
# @param listener_factory [#to, nil] injection point for tests, defaults to `Listen`
|
|
60
|
+
# @return [CredentialRefresher] a refresher with no sources when no credential file is
|
|
61
|
+
# configured, or when `credential_watch` is off
|
|
62
|
+
def from_config(configuration,
|
|
63
|
+
on_tls_change: -> { ActiveJob::Temporal.reload_client! },
|
|
64
|
+
logger: ActiveJob::Temporal::Logger,
|
|
65
|
+
listener_factory: nil)
|
|
66
|
+
sources = []
|
|
67
|
+
|
|
68
|
+
if configuration.credential_watch
|
|
69
|
+
tls = tls_paths(configuration)
|
|
70
|
+
sources << Source.new(name: "tls", paths: tls, on_change: on_tls_change) if tls.any?
|
|
71
|
+
|
|
72
|
+
token = configuration.api_key_file.to_s.strip
|
|
73
|
+
if token.present?
|
|
74
|
+
sources << Source.new(name: "api_key", paths: [token],
|
|
75
|
+
on_change: -> { ActiveJob::Temporal.refresh_api_key! })
|
|
76
|
+
end
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
new(
|
|
80
|
+
sources: sources,
|
|
81
|
+
poll_interval: configuration.credential_poll_interval,
|
|
82
|
+
file_events: configuration.credential_file_events,
|
|
83
|
+
logger: logger,
|
|
84
|
+
listener_factory: listener_factory
|
|
85
|
+
)
|
|
86
|
+
end
|
|
87
|
+
|
|
88
|
+
# @param configuration [Configuration] the gem configuration
|
|
89
|
+
# @return [Array<String>] configured TLS file paths, blanks removed
|
|
90
|
+
def tls_paths(configuration)
|
|
91
|
+
[
|
|
92
|
+
configuration.tls_cert_path,
|
|
93
|
+
configuration.tls_key_path,
|
|
94
|
+
configuration.tls_server_root_ca_cert_path
|
|
95
|
+
].compact.reject { |path| path.to_s.strip.empty? }
|
|
96
|
+
end
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
# @param sources [Array<Source>] credential groups to keep fresh
|
|
100
|
+
# @param poll_interval [Integer, Float] seconds between digest comparisons
|
|
101
|
+
# @param file_events [Boolean] wake the loop early on filesystem events, requires `listen`
|
|
102
|
+
# @param logger [#log_event, #warn, #error] structured logger
|
|
103
|
+
# @param listener_factory [#to, nil] injection point for tests, defaults to `Listen`
|
|
104
|
+
def initialize(sources:, poll_interval: DEFAULT_POLL_INTERVAL, file_events: false,
|
|
105
|
+
logger: ActiveJob::Temporal::Logger, listener_factory: nil)
|
|
106
|
+
@sources = sources.reject { |source| source.paths.compact.empty? }
|
|
107
|
+
@poll_interval = poll_interval
|
|
108
|
+
@file_events = file_events
|
|
109
|
+
@logger = logger
|
|
110
|
+
@listener_factory = listener_factory
|
|
111
|
+
@wakeups = Thread::Queue.new
|
|
112
|
+
@digests = {}
|
|
113
|
+
@thread = nil
|
|
114
|
+
@listener = nil
|
|
115
|
+
end
|
|
116
|
+
|
|
117
|
+
# Baselines each digest before polling, so a process booting on an already-rotated file does
|
|
118
|
+
# not fire a redundant reload on its first check.
|
|
119
|
+
#
|
|
120
|
+
# @return [self]
|
|
121
|
+
def start
|
|
122
|
+
return self if @sources.empty? || @thread
|
|
123
|
+
|
|
124
|
+
@sources.each { |source| @digests[source.name] = digest(source) }
|
|
125
|
+
@thread = Thread.new { run }
|
|
126
|
+
start_listener if @file_events
|
|
127
|
+
self
|
|
128
|
+
end
|
|
129
|
+
|
|
130
|
+
# @return [void]
|
|
131
|
+
def stop
|
|
132
|
+
@listener&.stop
|
|
133
|
+
@listener = nil
|
|
134
|
+
@wakeups.close
|
|
135
|
+
@thread&.join
|
|
136
|
+
@thread = nil
|
|
137
|
+
end
|
|
138
|
+
|
|
139
|
+
# Re-reads every source and fires the callbacks whose content changed. Safe to call at any
|
|
140
|
+
# rate: unchanged content is a no-op.
|
|
141
|
+
#
|
|
142
|
+
# @return [void]
|
|
143
|
+
def refresh_changed_sources
|
|
144
|
+
@sources.each { |source| refresh(source) }
|
|
145
|
+
end
|
|
146
|
+
|
|
147
|
+
# Wakes the poll loop so the next comparison happens now instead of on the next tick.
|
|
148
|
+
# Decides nothing on its own.
|
|
149
|
+
#
|
|
150
|
+
# @return [void]
|
|
151
|
+
def nudge
|
|
152
|
+
@wakeups << :wakeup
|
|
153
|
+
rescue ClosedQueueError
|
|
154
|
+
nil
|
|
155
|
+
end
|
|
156
|
+
|
|
157
|
+
private
|
|
158
|
+
|
|
159
|
+
def run
|
|
160
|
+
until @wakeups.closed?
|
|
161
|
+
@wakeups.pop(timeout: @poll_interval)
|
|
162
|
+
break if @wakeups.closed?
|
|
163
|
+
|
|
164
|
+
# A rotation emits a burst of events; collapse them into the single check that follows.
|
|
165
|
+
@wakeups.clear
|
|
166
|
+
refresh_changed_sources
|
|
167
|
+
end
|
|
168
|
+
end
|
|
169
|
+
|
|
170
|
+
def refresh(source)
|
|
171
|
+
current = digest(source)
|
|
172
|
+
return if current.nil? || current == @digests[source.name]
|
|
173
|
+
|
|
174
|
+
source.on_change.call
|
|
175
|
+
# Stamped only after the callback succeeds, so a failed reload is retried on the next tick.
|
|
176
|
+
@digests[source.name] = current
|
|
177
|
+
@logger.log_event("credential_refreshed", source: source.name)
|
|
178
|
+
rescue StandardError => e
|
|
179
|
+
@logger.error(
|
|
180
|
+
"credential_refresh_failed",
|
|
181
|
+
source: source.name,
|
|
182
|
+
error_class: e.class.name,
|
|
183
|
+
message: e.message
|
|
184
|
+
)
|
|
185
|
+
end
|
|
186
|
+
|
|
187
|
+
# A path can be briefly unreadable while kubelet swaps the mount. Treat that as "no reading"
|
|
188
|
+
# so the next tick retries, rather than reloading against a half-written credential.
|
|
189
|
+
def digest(source)
|
|
190
|
+
contents = source.paths.compact.map { |path| TLSFile.read(path) }
|
|
191
|
+
Digest::SHA256.hexdigest(contents.join("\0"))
|
|
192
|
+
rescue TLSFile::Error
|
|
193
|
+
nil
|
|
194
|
+
end
|
|
195
|
+
|
|
196
|
+
def start_listener
|
|
197
|
+
@listener = listener_factory.to(*watched_directories, ignore: KUBERNETES_VERSIONED_DIR) { nudge }
|
|
198
|
+
@listener.start
|
|
199
|
+
rescue LoadError => e
|
|
200
|
+
# Polling already guarantees the refresh, so a missing listen gem is a latency
|
|
201
|
+
# regression rather than a boot failure.
|
|
202
|
+
@logger.warn(
|
|
203
|
+
"credential_file_events_unavailable",
|
|
204
|
+
error_class: e.class.name,
|
|
205
|
+
message: e.message,
|
|
206
|
+
poll_interval_seconds: @poll_interval
|
|
207
|
+
)
|
|
208
|
+
end
|
|
209
|
+
|
|
210
|
+
def watched_directories
|
|
211
|
+
@sources.flat_map(&:paths).compact.map { |path| File.dirname(File.expand_path(path)) }.uniq
|
|
212
|
+
end
|
|
213
|
+
|
|
214
|
+
def listener_factory
|
|
215
|
+
@listener_factory ||= begin
|
|
216
|
+
require "listen"
|
|
217
|
+
Listen
|
|
218
|
+
end
|
|
219
|
+
end
|
|
220
|
+
end
|
|
221
|
+
end
|
|
222
|
+
end
|
|
@@ -129,11 +129,10 @@ en:
|
|
|
129
129
|
tls_domain:
|
|
130
130
|
blank: "must be present when configured"
|
|
131
131
|
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
requires_paths: "requires at least one TLS certificate path to watch"
|
|
132
|
+
tls:
|
|
133
|
+
requires_explicit_choice: "must be set explicitly to true, false, or TLS options when api_key or api_key_file is configured - the Temporal SDK enables TLS on its own whenever an API key is present, which fails the handshake against a plaintext frontend"
|
|
135
134
|
|
|
136
|
-
|
|
135
|
+
reload_signal:
|
|
137
136
|
blank: "must be present when configured"
|
|
138
137
|
invalid: "must be a signal name like HUP or USR1, got: %{value}"
|
|
139
138
|
|
|
@@ -141,9 +140,14 @@ en:
|
|
|
141
140
|
invalid_path: "must be a non-empty file path, got: %{value}"
|
|
142
141
|
unreadable_path: "must resolve to a readable regular file, got: %{value}"
|
|
143
142
|
|
|
144
|
-
|
|
143
|
+
credential_watch:
|
|
144
|
+
not_boolean: "must be true or false, got: %{value}"
|
|
145
|
+
|
|
146
|
+
credential_poll_interval:
|
|
147
|
+
invalid: "must be a positive number of seconds, got: %{value}"
|
|
148
|
+
|
|
149
|
+
credential_file_events:
|
|
145
150
|
not_boolean: "must be true or false, got: %{value}"
|
|
146
|
-
requires_path: "requires api_key_file to watch"
|
|
147
151
|
|
|
148
152
|
worker_activejob_workloads:
|
|
149
153
|
not_boolean: "must be true or false, got: %{value}"
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
require_relative "../temporal"
|
|
4
4
|
require_relative "rails_environment_loader"
|
|
5
|
-
require_relative "
|
|
5
|
+
require_relative "credential_refresher"
|
|
6
6
|
require_relative "reload_signal_queue"
|
|
7
7
|
require_relative "worker_client_reloader"
|
|
8
8
|
require_relative "worker_pool"
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: activejob-temporal
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.5.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Temporal Technologies
|
|
@@ -309,7 +309,6 @@ files:
|
|
|
309
309
|
- lib/activejob/temporal/cancel.rb
|
|
310
310
|
- lib/activejob/temporal/cancel/batch_canceller.rb
|
|
311
311
|
- lib/activejob/temporal/cancel/batch_summary.rb
|
|
312
|
-
- lib/activejob/temporal/certificate_watcher.rb
|
|
313
312
|
- lib/activejob/temporal/chain_options.rb
|
|
314
313
|
- lib/activejob/temporal/child_workflow_options.rb
|
|
315
314
|
- lib/activejob/temporal/client.rb
|
|
@@ -318,6 +317,7 @@ files:
|
|
|
318
317
|
- lib/activejob/temporal/configuration.rb
|
|
319
318
|
- lib/activejob/temporal/configured_job_compatibility.rb
|
|
320
319
|
- lib/activejob/temporal/connection_worker_pool.rb
|
|
320
|
+
- lib/activejob/temporal/credential_refresher.rb
|
|
321
321
|
- lib/activejob/temporal/dead_letter_payload_validation.rb
|
|
322
322
|
- lib/activejob/temporal/dead_letter_queue.rb
|
|
323
323
|
- lib/activejob/temporal/dependency_options.rb
|
|
@@ -1,116 +0,0 @@
|
|
|
1
|
-
# frozen_string_literal: true
|
|
2
|
-
|
|
3
|
-
module ActiveJob
|
|
4
|
-
module Temporal
|
|
5
|
-
# Watches TLS certificate files and runs a reload callback when they change.
|
|
6
|
-
class CertificateWatcher
|
|
7
|
-
DEFAULT_DEBOUNCE_SECONDS = 1.0
|
|
8
|
-
|
|
9
|
-
def self.paths_from_config(configuration)
|
|
10
|
-
[
|
|
11
|
-
configuration.tls_cert_path,
|
|
12
|
-
configuration.tls_key_path,
|
|
13
|
-
configuration.tls_server_root_ca_cert_path
|
|
14
|
-
].compact.reject { |path| path.to_s.strip.empty? }
|
|
15
|
-
end
|
|
16
|
-
|
|
17
|
-
def initialize(paths:, reload_callback:, listener_factory: nil, debounce_seconds: DEFAULT_DEBOUNCE_SECONDS)
|
|
18
|
-
@paths = paths.map { |path| File.expand_path(path) }.uniq
|
|
19
|
-
@reload_callback = reload_callback
|
|
20
|
-
@listener_factory = listener_factory
|
|
21
|
-
@debounce_seconds = debounce_seconds
|
|
22
|
-
@mutex = Mutex.new
|
|
23
|
-
@last_reload_at = nil
|
|
24
|
-
@listener = nil
|
|
25
|
-
@trailing_reload = nil
|
|
26
|
-
end
|
|
27
|
-
|
|
28
|
-
def start
|
|
29
|
-
return self if @paths.empty? || @listener
|
|
30
|
-
|
|
31
|
-
@listener = listener_factory.to(*directories) do |modified, added, removed|
|
|
32
|
-
handle_changes(modified + added + removed)
|
|
33
|
-
end
|
|
34
|
-
@listener.start
|
|
35
|
-
self
|
|
36
|
-
end
|
|
37
|
-
|
|
38
|
-
def stop
|
|
39
|
-
@listener&.stop
|
|
40
|
-
@listener = nil
|
|
41
|
-
@mutex.synchronize { @trailing_reload }&.kill
|
|
42
|
-
end
|
|
43
|
-
|
|
44
|
-
def handle_changes(changed_paths)
|
|
45
|
-
return unless relevant_change?(changed_paths)
|
|
46
|
-
|
|
47
|
-
if debounced?
|
|
48
|
-
schedule_trailing_reload
|
|
49
|
-
else
|
|
50
|
-
reload(retry_on_failure: true)
|
|
51
|
-
end
|
|
52
|
-
end
|
|
53
|
-
|
|
54
|
-
private
|
|
55
|
-
|
|
56
|
-
# A failed reload clears the debounce stamp so the next change reloads
|
|
57
|
-
# immediately, and gets one retry in case no further change arrives.
|
|
58
|
-
def reload(retry_on_failure:)
|
|
59
|
-
@reload_callback.call
|
|
60
|
-
rescue StandardError
|
|
61
|
-
@mutex.synchronize { @last_reload_at = nil }
|
|
62
|
-
schedule_trailing_reload if retry_on_failure
|
|
63
|
-
end
|
|
64
|
-
|
|
65
|
-
# Cert and key rotate as separate writes, so the second one lands inside
|
|
66
|
-
# the debounce window. Flush it once the window closes instead of dropping it.
|
|
67
|
-
def schedule_trailing_reload
|
|
68
|
-
@mutex.synchronize do
|
|
69
|
-
return if @trailing_reload&.alive?
|
|
70
|
-
|
|
71
|
-
@trailing_reload = Thread.new do
|
|
72
|
-
sleep(@debounce_seconds)
|
|
73
|
-
@mutex.synchronize { @last_reload_at = Process.clock_gettime(Process::CLOCK_MONOTONIC) }
|
|
74
|
-
reload(retry_on_failure: false)
|
|
75
|
-
end
|
|
76
|
-
end
|
|
77
|
-
end
|
|
78
|
-
|
|
79
|
-
def directories
|
|
80
|
-
@directories ||= @paths.map { |path| File.dirname(path) }.uniq
|
|
81
|
-
end
|
|
82
|
-
|
|
83
|
-
def listener_factory
|
|
84
|
-
@listener_factory ||= begin
|
|
85
|
-
require "listen"
|
|
86
|
-
Listen
|
|
87
|
-
rescue LoadError => e
|
|
88
|
-
raise LoadError, "listen gem is required when tls_cert_watch is enabled: #{e.message}"
|
|
89
|
-
end
|
|
90
|
-
end
|
|
91
|
-
|
|
92
|
-
# Kubernetes projected and secret volumes rotate atomically: kubelet writes a new timestamped
|
|
93
|
-
# directory and swaps a `..data` symlink, so the watched file's own path never appears in the
|
|
94
|
-
# change events. Any change inside a watched file's directory therefore counts as relevant;
|
|
95
|
-
# the debounce absorbs the multi-event burst a rotation produces.
|
|
96
|
-
def relevant_change?(changed_paths)
|
|
97
|
-
changed_paths.any? do |path|
|
|
98
|
-
expanded = File.expand_path(path)
|
|
99
|
-
@paths.include?(expanded) || directories.any? { |dir| expanded.start_with?("#{dir}/") }
|
|
100
|
-
end
|
|
101
|
-
end
|
|
102
|
-
|
|
103
|
-
def debounced?
|
|
104
|
-
return false unless @debounce_seconds.positive?
|
|
105
|
-
|
|
106
|
-
now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
107
|
-
@mutex.synchronize do
|
|
108
|
-
return true if @last_reload_at && (now - @last_reload_at) < @debounce_seconds
|
|
109
|
-
|
|
110
|
-
@last_reload_at = now
|
|
111
|
-
false
|
|
112
|
-
end
|
|
113
|
-
end
|
|
114
|
-
end
|
|
115
|
-
end
|
|
116
|
-
end
|