quonfig 1.1.0 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +12 -0
- data/README.md +36 -0
- data/lib/quonfig/client.rb +61 -7
- data/lib/quonfig/config_loader.rb +39 -6
- data/lib/quonfig/http_connection.rb +37 -6
- data/lib/quonfig/options.rb +11 -1
- data/lib/quonfig/sse_config_client.rb +24 -15
- data/lib/quonfig/telemetry/failover_aggregator.rb +113 -0
- data/lib/quonfig/telemetry/telemetry_reporter.rb +11 -1
- data/lib/quonfig/version.rb +1 -1
- data/lib/quonfig.rb +1 -0
- metadata +3 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: d9da26b9a9f898ec76bf8520043abc42db011adf5e298cffa16ee8764c2c0ed6
|
|
4
|
+
data.tar.gz: '0189ef6ee1e93cec985f9f5b1f1b81f833527bf3c22a1af7886778d8f2dcb50e'
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: e363cf1a35bbc428f35d1d4cebea4223dc84c4a1ddd39c3a7dcbc22bafb58a757b5fde92f357dd1a9b8c0e0cc90db247addd88c873e19887cd1d0eae0bc5332b
|
|
7
|
+
data.tar.gz: e81f8441d30449624e3c64b506e113d386ce8a55656943e4523cb2d919d97aad5d8afd68df6462df1324cb2368dd8d9694e808146d622dd790ccf722e630e442
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,17 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 1.2.0 - 2026-07-08
|
|
4
|
+
|
|
5
|
+
- **Feat (options): warn when an explicit `api_urls` disables automatic failover (qfg-41nh.26).** The default (and every `QUONFIG_DOMAIN`-derived) `api_urls` list carries a primary AND a secondary leg, and the HTTP config-fetch hedges/fails over between them (the secondary runs on separate infrastructure). An explicit `api_urls:` replaces that list wholesale, so a single-entry override silently dropped the secondary and disabled automatic failover. The client now logs one WARN at init when the caller explicitly set `api_urls` and the resolved list has fewer than two legs, pointing at the fix (pass both a primary and a secondary URL). Behavior is otherwise unchanged; no new dependencies. A new README "Failover & `QUONFIG_DOMAIN`" section documents the URL derivation and the failover model.
|
|
6
|
+
- **Feat (telemetry): emit failover-behavior signals on the existing telemetry wire (qfg-41nh.18).** The SDK now folds three additive failover counters into the periodic telemetry flush so the failover dashboards can be built — previously no SDK emitted any failover signal. Each flush window that saw failover activity ships one `failover` event alongside the existing `summaries` / `contextShapes` / `exampleContexts` events: `hedgeFired` (config-fetch cycles where the parallel primary+secondary hedge fired its secondary leg), `guardRejected` (installs dropped by the reject-older ordering guard, counted on BOTH the HTTP config-fetch path and the SSE message path), and `resolvedFromPrimary` / `resolvedFromSecondary` (which upstream leg served each successful HTTP install; SSE/datadir installs are not counted). `resolvedFromLkg` is reserved and always `0`. The event is emitted only when at least one counter is non-zero in the window, so a steady-state client with no config churn emits nothing. Recording rides the existing telemetry opt-out (nothing is emitted unless telemetry is enabled) and adds only a per-config-refresh mutex increment — no new dependencies, no per-evaluation overhead. Wire shape is additive and matches sdk-go exactly (an older api-telemetry strips the unknown field).
|
|
7
|
+
|
|
8
|
+
## 1.1.1 - 2026-07-03
|
|
9
|
+
|
|
10
|
+
- **Fix (SSE): the stream leg is pinned to the primary stream URL — SSE never fails over (qfg-41nh.6).** The reconnect loop previously rotated through `sse_api_urls` on every reconnect, so with the default two derived stream URLs a primary blip silently parked the live stream on `stream.secondary` and bound config freshness to the mirror. The stream now always dials `sse_api_urls[0]` and retries it forever with backoff (matching sdk-go); failover remains HTTP-poll only, and the HTTP config-fetch hedge still uses both `api_urls` legs. The `sse_failed_over_to_secondary?` diagnostic is now latched at the dial site, and the failover chaos harness hands the client both stream legs so the f05 probe actually exercises the pin.
|
|
11
|
+
- **Fix (delivery): per-leg config-fetch aborts are now wall-clock (qfg-41nh.6).** The per-leg timeout was enforced with Faraday/Net::HTTP phase timeouts, which are per-read — a slow-drip upstream (one byte per interval) could hold a "bounded" leg open indefinitely and wedge the init fetch, a fallback poll tick, or the hedge drain. `timeout_ms` (plus a small headroom) is now also enforced as a wall-clock ceiling over the whole request, surfacing a `Faraday::TimeoutError` through the existing failure paths. Hedged legs additionally always settle their drain slot, so `fetch!` is guaranteed to return within the per-leg abort budget.
|
|
12
|
+
- **Fix (client): liveness signals no longer lie during outages (qfg-41nh.6).** The fallback poll worker stamped `last_successful_refresh` and fired `on_update` on every tick regardless of fetch outcome, so a total outage with `on_init_failure: :return` reported `ready? == true` over an empty store. The worker now stamps freshness only on successful fetches (`:updated` / `:not_modified`; a 304 or guard-rejected-but-successful response counts) and fires `on_update` only when an envelope is actually installed, matching sdk-go.
|
|
13
|
+
- **Fix (client): the fallback poller fetches immediately on engage and logs the engage edge (qfg-41nh.6).** The poll worker previously slept a full interval before its first fetch, so the first post-engage data arrived ~180s after SSE loss (default 60s interval) vs ~120s in sdk-go; it now fetches first, then sleeps. Engaging Layer 2 also logs at WARN (previously silent — only the stop path logged, at debug), matching sdk-go's operator signal.
|
|
14
|
+
|
|
3
15
|
## 1.1.0 - 2026-07-01
|
|
4
16
|
|
|
5
17
|
- **Feat (delivery): the HTTP config-fetch is now a parallel-failover hedge (qfg-7h5d.1.14).** On every init/refresh config fetch the SDK fires the **primary** `api_urls` leg first; if it answers within `config_fetch_hedge_delay_ms` it **wins and the secondary is never contacted** (cold standby — a healthy system adds zero secondary load). If the primary is slow past the hedge delay **or** errors fast, the SDK **also** fires the secondary **in parallel** without cancelling the primary. Whatever arrives is installed through the existing reject-older guard, so watermark-max falls out for free: the higher generation wins, a late older payload never regresses an established client, and a late newer payload heals forward. Readiness latches on the first successful install; a late-but-newer leg heals forward afterward. SSE is untouched — only the HTTP config-fetch path hedges. Both legs failing preserves the existing init-failure semantics (`on_init_failure`).
|
data/README.md
CHANGED
|
@@ -258,6 +258,42 @@ Quonfig::Client.new(
|
|
|
258
258
|
| `data_dir_auto_reload_debounce_ms` | `Integer` (ms) | `200` | Debounce window for the auto-reload watcher — events arriving inside the window are coalesced into a single re-read. Ignored when `data_dir_auto_reload` is `false`. |
|
|
259
259
|
| `logger` | Logger-like object | `nil` | Optional host-app logger (e.g. `Rails.logger`). Must respond to `debug`/`info`/`warn`/`error`. When set, all SDK warnings/errors flow through this logger instead of the default stderr / SemanticLogger backend. |
|
|
260
260
|
|
|
261
|
+
## Failover & `QUONFIG_DOMAIN`
|
|
262
|
+
|
|
263
|
+
By default the SDK derives every hostname from `QUONFIG_DOMAIN` (default
|
|
264
|
+
`quonfig.com`):
|
|
265
|
+
|
|
266
|
+
| Role | URL |
|
|
267
|
+
|--------------------------|-----------------------------------------|
|
|
268
|
+
| Config fetch (primary) | `https://primary.quonfig.com` |
|
|
269
|
+
| SSE stream (primary) | `https://stream.primary.quonfig.com` |
|
|
270
|
+
| Config fetch (secondary) | `https://secondary.quonfig.com` |
|
|
271
|
+
| SSE stream (secondary) | `https://stream.secondary.quonfig.com` |
|
|
272
|
+
| Telemetry | `https://telemetry.quonfig.com` |
|
|
273
|
+
|
|
274
|
+
Set `QUONFIG_DOMAIN` to move all of them together (e.g.
|
|
275
|
+
`QUONFIG_DOMAIN=quonfig-staging.com`). **Automatic failover and hedging between
|
|
276
|
+
the primary and the secondary are on by default** — the secondary runs on
|
|
277
|
+
separate infrastructure, and the HTTP config-fetch fails over to it if the
|
|
278
|
+
primary is unreachable and hedges to it if the primary is slow.
|
|
279
|
+
|
|
280
|
+
`api_urls:` replaces the derived list wholesale. To keep automatic failover
|
|
281
|
+
with custom URLs, **pass both a primary and a secondary URL**:
|
|
282
|
+
|
|
283
|
+
```ruby
|
|
284
|
+
Quonfig::Client.new(
|
|
285
|
+
sdk_key: 'your-sdk-key',
|
|
286
|
+
api_urls: [
|
|
287
|
+
'https://primary.your-proxy.example',
|
|
288
|
+
'https://secondary.your-proxy.example'
|
|
289
|
+
]
|
|
290
|
+
)
|
|
291
|
+
```
|
|
292
|
+
|
|
293
|
+
A single URL disables failover, and the SDK logs a warning at init. See
|
|
294
|
+
https://docs.quonfig.com/docs/explanations/architecture/resiliency for the full
|
|
295
|
+
model.
|
|
296
|
+
|
|
261
297
|
## Typed getters
|
|
262
298
|
|
|
263
299
|
Each typed getter takes a config key and an optional context hash. If the key
|
data/lib/quonfig/client.rb
CHANGED
|
@@ -66,6 +66,16 @@ module Quonfig
|
|
|
66
66
|
@poll_supervisor = nil
|
|
67
67
|
@stopped = false
|
|
68
68
|
@telemetry_reporter = nil
|
|
69
|
+
# Shared failover-telemetry aggregator (qfg-41nh.18). Created once and
|
|
70
|
+
# shared between the ConfigLoader (which records hedge-fired /
|
|
71
|
+
# guard-rejected / resolved-from at the failover call sites) and the
|
|
72
|
+
# TelemetryReporter (which drains it on each flush). Created before the
|
|
73
|
+
# ConfigLoader so the initial HTTP fetch — often the ONLY HTTP fetch when
|
|
74
|
+
# SSE is healthy — is captured. Recording is a per-config-refresh mutex +
|
|
75
|
+
# increment (negligible); nothing is EMITTED unless telemetry is enabled
|
|
76
|
+
# (the reporter, which owns the drain, only starts then). Survives fork:
|
|
77
|
+
# the pre-fork stop drains it, the child's rebuilt reporter drains onward.
|
|
78
|
+
@failover_aggregator = Quonfig::Telemetry::FailoverAggregator.new
|
|
69
79
|
@state_mutex = Mutex.new
|
|
70
80
|
@last_successful_refresh = nil
|
|
71
81
|
@sse_state = :idle
|
|
@@ -689,6 +699,7 @@ module Quonfig
|
|
|
689
699
|
context_shape_aggregator: shape_aggregator,
|
|
690
700
|
example_contexts_aggregator: example_aggregator,
|
|
691
701
|
evaluation_summaries_aggregator: summaries_aggregator,
|
|
702
|
+
failover_aggregator: @failover_aggregator,
|
|
692
703
|
sync_interval: @options.collect_sync_interval
|
|
693
704
|
)
|
|
694
705
|
|
|
@@ -813,8 +824,9 @@ module Quonfig
|
|
|
813
824
|
|
|
814
825
|
warn_if_pin_ignored_in_delivery_mode
|
|
815
826
|
warn_if_hedge_abort_exceeds_init_timeout
|
|
827
|
+
warn_if_explicit_api_urls_disables_failover
|
|
816
828
|
|
|
817
|
-
@config_loader = Quonfig::ConfigLoader.new(@store, @options)
|
|
829
|
+
@config_loader = Quonfig::ConfigLoader.new(@store, @options, failover_aggregator: @failover_aggregator)
|
|
818
830
|
|
|
819
831
|
perform_initial_fetch
|
|
820
832
|
|
|
@@ -914,6 +926,27 @@ module Quonfig
|
|
|
914
926
|
)
|
|
915
927
|
end
|
|
916
928
|
|
|
929
|
+
# qfg-41nh.26: the SDK's default (and every QUONFIG_DOMAIN-derived) api_urls
|
|
930
|
+
# list carries a primary AND a secondary leg, and the HTTP config-fetch
|
|
931
|
+
# hedges/fails over between them (secondary runs on separate infra). An
|
|
932
|
+
# explicit `api_urls:` replaces that list wholesale, so a single-entry
|
|
933
|
+
# override silently drops the secondary and disables automatic failover.
|
|
934
|
+
# Warn once at init in delivery mode so the customer isn't surprised.
|
|
935
|
+
# Mirrors sdk-go's construction-time warning (quonfig.go). Does not change
|
|
936
|
+
# behavior. Fired only when the caller explicitly set api_urls AND the
|
|
937
|
+
# resolved list has fewer than two legs — never on the default/derived
|
|
938
|
+
# two-URL list.
|
|
939
|
+
def warn_if_explicit_api_urls_disables_failover
|
|
940
|
+
return unless @options.respond_to?(:api_urls_explicit)
|
|
941
|
+
return unless @options.api_urls_explicit
|
|
942
|
+
return unless Array(@options.config_api_urls).length < 2
|
|
943
|
+
|
|
944
|
+
LOG.warn(
|
|
945
|
+
'[quonfig] explicit api_urls disables automatic failover to the ' \
|
|
946
|
+
'secondary; pass both primary and secondary URLs to keep it'
|
|
947
|
+
)
|
|
948
|
+
end
|
|
949
|
+
|
|
917
950
|
def handle_init_failure(err)
|
|
918
951
|
if @options.on_init_failure == Quonfig::Options::ON_INITIALIZATION_FAILURE::RETURN
|
|
919
952
|
LOG.warn "[quonfig] Initialization did not complete cleanly; continuing with empty store: #{err.message}"
|
|
@@ -970,18 +1003,35 @@ module Quonfig
|
|
|
970
1003
|
|
|
971
1004
|
poll_seconds = poll_ms / 1000.0
|
|
972
1005
|
stopped_ref = -> { @stopped }
|
|
1006
|
+
# Fetch-first (qfg-41nh.6): the poller fetches IMMEDIATELY on engage and
|
|
1007
|
+
# then sleeps the interval — matching sdk-go's fallback_poller.go
|
|
1008
|
+
# engage() (immediate Fetch, then ticker). A sleep-first worker made the
|
|
1009
|
+
# first post-engage data arrive a full extra interval late (~180s after
|
|
1010
|
+
# SSE loss with the default 60s interval, vs ~120s in go).
|
|
973
1011
|
worker = lambda do |notify_delivered|
|
|
974
1012
|
loop do
|
|
975
1013
|
break if stopped_ref.call
|
|
976
1014
|
|
|
977
|
-
|
|
1015
|
+
result = @config_loader.fetch!
|
|
1016
|
+
# Liveness honesty (qfg-41nh.6): stamp freshness only when the fetch
|
|
1017
|
+
# actually SUCCEEDED. :updated and :not_modified (a 304, or a
|
|
1018
|
+
# guard-rejected-but-successful response) are successes — the
|
|
1019
|
+
# channel is healthy; :failed must NOT advance
|
|
1020
|
+
# last_successful_refresh (pre-fix every tick stamped + notified
|
|
1021
|
+
# regardless of outcome, so ready?/freshness reported healthy over
|
|
1022
|
+
# an empty store through a total outage). on_update fires only on a
|
|
1023
|
+
# real install, matching sdk-go (OnConfigUpdate fires in
|
|
1024
|
+
# installEnvelope only).
|
|
1025
|
+
if result != :failed
|
|
1026
|
+
sync_evaluator_env_id!
|
|
1027
|
+
record_refresh!
|
|
1028
|
+
notify_delivered.call
|
|
1029
|
+
end
|
|
1030
|
+
notify_on_update_callback if result == :updated
|
|
1031
|
+
|
|
978
1032
|
break if stopped_ref.call
|
|
979
1033
|
|
|
980
|
-
|
|
981
|
-
sync_evaluator_env_id!
|
|
982
|
-
record_refresh!
|
|
983
|
-
notify_delivered.call
|
|
984
|
-
notify_on_update_callback
|
|
1034
|
+
sleep poll_seconds
|
|
985
1035
|
end
|
|
986
1036
|
end
|
|
987
1037
|
|
|
@@ -990,6 +1040,10 @@ module Quonfig
|
|
|
990
1040
|
)
|
|
991
1041
|
@state_mutex.synchronize { @poll_supervisor = supervisor }
|
|
992
1042
|
supervisor.start
|
|
1043
|
+
# Operator signal (qfg-41nh.6): engage was previously unlogged (only the
|
|
1044
|
+
# stop path logged, at debug). Matches sdk-go's WARN on OnEngage — the
|
|
1045
|
+
# poller engaging means the SDK is in a degraded update mode.
|
|
1046
|
+
LOG.warn "[quonfig] Layer 2 fallback poller engaged (interval=#{poll_ms}ms)"
|
|
993
1047
|
end
|
|
994
1048
|
|
|
995
1049
|
# Invoke the customer-supplied on_update callback under a rescue. A raise
|
|
@@ -37,7 +37,7 @@ module Quonfig
|
|
|
37
37
|
#
|
|
38
38
|
# Backward compat: callers that pass a single +base_client+ (mock client
|
|
39
39
|
# used by tests that expects `.options`) are still supported.
|
|
40
|
-
def initialize(store_or_base_client, options = nil, logger: nil)
|
|
40
|
+
def initialize(store_or_base_client, options = nil, logger: nil, failover_aggregator: nil)
|
|
41
41
|
if options.nil? && store_or_base_client.respond_to?(:options)
|
|
42
42
|
# Legacy shape: ConfigLoader.new(base_client)
|
|
43
43
|
@options = store_or_base_client.options
|
|
@@ -47,6 +47,11 @@ module Quonfig
|
|
|
47
47
|
@options = options
|
|
48
48
|
end
|
|
49
49
|
|
|
50
|
+
# Optional failover-telemetry sink (qfg-41nh.18). When telemetry is
|
|
51
|
+
# enabled the Client passes the shared FailoverAggregator; nil otherwise,
|
|
52
|
+
# so every record call below is a no-op under a telemetry opt-out.
|
|
53
|
+
@failover_aggregator = failover_aggregator
|
|
54
|
+
|
|
50
55
|
@api_config = Concurrent::Map.new
|
|
51
56
|
# qfg-7h5d.1.14: per-leg ETag is load-bearing for the parallel hedge. The
|
|
52
57
|
# hedge runs the primary and secondary legs concurrently; a SINGLE shared
|
|
@@ -179,16 +184,26 @@ module Quonfig
|
|
|
179
184
|
# mutually exclusive — exactly one of suppress/fire wins.
|
|
180
185
|
gate = Mutex.new
|
|
181
186
|
secondary_fired = false
|
|
182
|
-
|
|
187
|
+
# Distinct from secondary_fired (which is ALSO set true by suppress on a
|
|
188
|
+
# fast primary win): this is set only when the secondary leg was ACTUALLY
|
|
189
|
+
# spawned, so it is the exact "the hedge fired" signal (qfg-41nh.18).
|
|
190
|
+
secondary_spawned = false
|
|
191
|
+
|
|
192
|
+
# Each leg is wall-clock bounded by abort_ms (HttpConnection enforces a
|
|
193
|
+
# whole-request deadline, qfg-41nh.6), so a leg ALWAYS settles — a
|
|
194
|
+
# slow-drip upstream can no longer hold a leg open past the abort and
|
|
195
|
+
# wedge the drain below. The push lives in an ensure so even a
|
|
196
|
+
# non-StandardError escape still settles this leg's drain slot.
|
|
183
197
|
run_leg = lambda do |index|
|
|
184
198
|
Thread.new do
|
|
185
|
-
result =
|
|
186
|
-
|
|
199
|
+
result = :failed
|
|
200
|
+
begin
|
|
201
|
+
result = fetch_from(urls[index], index, timeout_ms: abort_ms)
|
|
187
202
|
rescue StandardError => e
|
|
188
203
|
@logger.debug "Hedge leg #{index} failed: #{e.class}: #{e.message}"
|
|
189
|
-
|
|
204
|
+
ensure
|
|
205
|
+
results.push([:done, index, result])
|
|
190
206
|
end
|
|
191
|
-
results.push([:done, index, result])
|
|
192
207
|
end
|
|
193
208
|
end
|
|
194
209
|
|
|
@@ -197,6 +212,7 @@ module Quonfig
|
|
|
197
212
|
next false if secondary_fired
|
|
198
213
|
|
|
199
214
|
secondary_fired = true
|
|
215
|
+
secondary_spawned = true
|
|
200
216
|
end
|
|
201
217
|
run_leg.call(1) if spawn
|
|
202
218
|
end
|
|
@@ -250,6 +266,14 @@ module Quonfig
|
|
|
250
266
|
# Stop the timer if it is still sleeping (already-fired is harmless).
|
|
251
267
|
hedge_timer.kill if hedge_timer.alive?
|
|
252
268
|
|
|
269
|
+
# Failover observability (qfg-41nh.18): if the secondary leg was actually
|
|
270
|
+
# spawned this cycle, the hedge fired (the primary was slow past the hedge
|
|
271
|
+
# delay or errored fast). Recorded once per cycle regardless of which leg's
|
|
272
|
+
# payload won the reject-older guard. Read under the gate — the flag is
|
|
273
|
+
# written under it by fire_secondary, which may run on the timer thread.
|
|
274
|
+
hedged = gate.synchronize { secondary_spawned }
|
|
275
|
+
@failover_aggregator&.record_hedge_fired if hedged
|
|
276
|
+
|
|
253
277
|
return :updated if installed
|
|
254
278
|
return :not_modified if saw_not_modified
|
|
255
279
|
|
|
@@ -363,6 +387,10 @@ module Quonfig
|
|
|
363
387
|
# on stale config would be worse (mirrors sdk-node).
|
|
364
388
|
unless @held_generation.nil? || incoming_gen <= 0 || incoming_gen > @held_generation
|
|
365
389
|
@logger.debug "Reject-older guard: dropping incoming generation #{incoming_gen} <= held #{@held_generation} (source=#{source})"
|
|
390
|
+
# Failover observability (qfg-41nh.18): count the guard rejection. This
|
|
391
|
+
# single guard covers BOTH the HTTP config-fetch path and the SSE
|
|
392
|
+
# message path (apply_envelope) — every network install funnels here.
|
|
393
|
+
@failover_aggregator&.record_guard_rejected
|
|
366
394
|
return :not_modified
|
|
367
395
|
end
|
|
368
396
|
|
|
@@ -382,6 +410,11 @@ module Quonfig
|
|
|
382
410
|
@held_generation = incoming_gen
|
|
383
411
|
@install_count += 1
|
|
384
412
|
@resolved_from_index = source_index unless source_index.nil?
|
|
413
|
+
# Failover observability (qfg-41nh.18): record which leg served this
|
|
414
|
+
# successful HTTP install (source_index 0 = primary, > 0 = secondary).
|
|
415
|
+
# SSE / datadir installs pass source_index nil and are ignored by the
|
|
416
|
+
# aggregator, so they are never counted.
|
|
417
|
+
@failover_aggregator&.record_resolved_from(source_index)
|
|
385
418
|
|
|
386
419
|
# Replace the live store atomically.
|
|
387
420
|
return if @store.nil?
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
require 'base64'
|
|
4
4
|
require 'json'
|
|
5
|
+
require 'timeout'
|
|
5
6
|
|
|
6
7
|
module Quonfig
|
|
7
8
|
class HttpConnection
|
|
@@ -13,13 +14,26 @@ module Quonfig
|
|
|
13
14
|
'X-Quonfig-SDK-Version' => SDK_VERSION
|
|
14
15
|
}.freeze
|
|
15
16
|
|
|
17
|
+
# qfg-41nh.6 (WS2.4): headroom added to +timeout_ms+ to form the WALL-CLOCK
|
|
18
|
+
# ceiling for the whole request. Faraday's open/read timeouts are per-phase
|
|
19
|
+
# and — on the Net::HTTP adapter — per-READ: every byte that arrives resets
|
|
20
|
+
# the read deadline, so a slow-drip upstream (one byte per interval) holds a
|
|
21
|
+
# "bounded" request open indefinitely and wedges the caller (init fetch,
|
|
22
|
+
# fallback poll tick, hedge drain). The wall clock is the true per-leg abort
|
|
23
|
+
# (sdk-go's per-leg context deadline is wall-clock). The headroom keeps
|
|
24
|
+
# Faraday's own, more specific phase timeouts winning the simple-stall race;
|
|
25
|
+
# the wall clock only fires on drip-feed pathologies that per-read deadlines
|
|
26
|
+
# structurally cannot catch.
|
|
27
|
+
WALL_CLOCK_HEADROOM_S = 0.25
|
|
28
|
+
|
|
16
29
|
# +timeout_ms+ (qfg-7h5d.1.9): per-request bound applied to BOTH the connect
|
|
17
|
-
# (open) and read phases of every request made through this connection
|
|
30
|
+
# (open) and read phases of every request made through this connection, AND
|
|
31
|
+
# (qfg-41nh.6) enforced as a wall-clock ceiling over the whole request. nil
|
|
18
32
|
# leaves Faraday's defaults (no timeout) in place — preserving the prior
|
|
19
33
|
# behavior for callers that don't pass one. The config-fetch path passes
|
|
20
|
-
# Options#config_fetch_timeout_ms
|
|
21
|
-
#
|
|
22
|
-
# whole init budget.
|
|
34
|
+
# Options#config_fetch_timeout_ms (sequential) or the hedge abort (hedged
|
|
35
|
+
# legs) so a hung OR drip-feeding upstream aborts fast instead of blocking
|
|
36
|
+
# the caller's whole init budget.
|
|
23
37
|
def initialize(uri, sdk_key, timeout_ms: nil)
|
|
24
38
|
@uri = uri
|
|
25
39
|
@sdk_key = sdk_key
|
|
@@ -29,11 +43,11 @@ module Quonfig
|
|
|
29
43
|
attr_reader :uri
|
|
30
44
|
|
|
31
45
|
def get(path, headers = {})
|
|
32
|
-
connection(headers).get(path)
|
|
46
|
+
with_wall_clock_deadline { connection(headers).get(path) }
|
|
33
47
|
end
|
|
34
48
|
|
|
35
49
|
def post(path, body)
|
|
36
|
-
connection.post(path, body.to_json)
|
|
50
|
+
with_wall_clock_deadline { connection.post(path, body.to_json) }
|
|
37
51
|
end
|
|
38
52
|
|
|
39
53
|
def connection(headers = {})
|
|
@@ -54,6 +68,23 @@ module Quonfig
|
|
|
54
68
|
|
|
55
69
|
private
|
|
56
70
|
|
|
71
|
+
# Enforce +timeout_ms+ (+ headroom) as a wall-clock deadline over the whole
|
|
72
|
+
# request. Timeout.timeout is the codebase's accepted backstop pattern (the
|
|
73
|
+
# init path wraps fetch! the same way in Client#perform_initial_fetch); a
|
|
74
|
+
# deadline check inside the read loop isn't practical here because Faraday
|
|
75
|
+
# 1.x's Net::HTTP adapter exposes no per-chunk streaming hook. Raises
|
|
76
|
+
# Faraday::TimeoutError so callers' existing timeout rescue paths apply
|
|
77
|
+
# unchanged. No-op when no timeout_ms was configured.
|
|
78
|
+
def with_wall_clock_deadline(&block)
|
|
79
|
+
return block.call unless @timeout_ms
|
|
80
|
+
|
|
81
|
+
deadline_s = (@timeout_ms / 1000.0) + WALL_CLOCK_HEADROOM_S
|
|
82
|
+
Timeout.timeout(deadline_s, Faraday::TimeoutError,
|
|
83
|
+
"wall-clock per-request deadline #{deadline_s}s exceeded for #{@uri}") do
|
|
84
|
+
block.call
|
|
85
|
+
end
|
|
86
|
+
end
|
|
87
|
+
|
|
57
88
|
def auth_header
|
|
58
89
|
"Basic #{Base64.strict_encode64("1:#{@sdk_key}")}"
|
|
59
90
|
end
|
data/lib/quonfig/options.rb
CHANGED
|
@@ -8,7 +8,7 @@ module Quonfig
|
|
|
8
8
|
attr_reader :sdk_key, :environment, :api_urls, :sse_api_urls, :telemetry_destination, :config_api_urls,
|
|
9
9
|
:on_no_default, :init_timeout_ms, :on_init_failure, :collect_sync_interval, :datadir, :enable_sse, :fallback_poll_enabled, :fallback_poll_interval_ms, :global_context, :logger_key, :logger, :enable_quonfig_user_context,
|
|
10
10
|
:data_dir_auto_reload, :data_dir_auto_reload_debounce_ms, :config_fetch_timeout_ms,
|
|
11
|
-
:config_fetch_hedge_delay_ms, :config_fetch_hedge_abort_ms
|
|
11
|
+
:config_fetch_hedge_delay_ms, :config_fetch_hedge_abort_ms, :api_urls_explicit
|
|
12
12
|
attr_accessor :is_fork
|
|
13
13
|
|
|
14
14
|
# Default fallback poll interval, in milliseconds. The SDK polls api-delivery
|
|
@@ -308,6 +308,16 @@ module Quonfig
|
|
|
308
308
|
# 3. Hardcoded DEFAULT_DOMAIN ('quonfig.com')
|
|
309
309
|
domain = Quonfig::Options.domain
|
|
310
310
|
|
|
311
|
+
# Whether the caller explicitly supplied api_urls (vs. the SDK deriving
|
|
312
|
+
# both legs from QUONFIG_DOMAIN / DEFAULT_DOMAIN). The default and every
|
|
313
|
+
# domain-derived list carries a primary AND a secondary leg, so the SDK
|
|
314
|
+
# hedges/fails over automatically; an explicit single-entry override
|
|
315
|
+
# silently drops the secondary. Client warns once at init when this is
|
|
316
|
+
# true and the resolved list has < 2 legs (see
|
|
317
|
+
# Client#warn_if_explicit_api_urls_disables_failover). Mirrors sdk-go's
|
|
318
|
+
# apiURLsExplicit flag (qfg-41nh.26).
|
|
319
|
+
@api_urls_explicit = !api_urls.nil?
|
|
320
|
+
|
|
311
321
|
@api_urls = Array(api_urls || Quonfig::Options.derive_api_urls(domain))
|
|
312
322
|
.map { |url| remove_trailing_slash(url) }
|
|
313
323
|
|
|
@@ -87,14 +87,16 @@ module Quonfig
|
|
|
87
87
|
@conn_mutex = Mutex.new
|
|
88
88
|
@active_http = nil
|
|
89
89
|
|
|
90
|
-
@source_index = -1
|
|
91
90
|
@last_event_id = nil
|
|
92
|
-
# qfg-7h5d.1.9: latches true if
|
|
93
|
-
# non-primary
|
|
94
|
-
# does NOT fail over to the secondary (f05) —
|
|
95
|
-
# and degrades via the single-upstream SSE↔HTTP fallback
|
|
96
|
-
#
|
|
97
|
-
#
|
|
91
|
+
# qfg-7h5d.1.9 / qfg-41nh.6: latches true if a connection attempt ever
|
|
92
|
+
# targets a non-primary sse_api_urls leg. The failover epic asserts SSE
|
|
93
|
+
# does NOT fail over to the secondary (f05) — the stream is PINNED to
|
|
94
|
+
# sse_api_urls[0] and degrades via the single-upstream SSE↔HTTP fallback,
|
|
95
|
+
# so with a correct #current_url this never flips even when a secondary
|
|
96
|
+
# stream URL is configured (options derive one from api_urls by default).
|
|
97
|
+
# The latch is measured at the dial site in #stream_once, so a regression
|
|
98
|
+
# that reintroduces stream-URL rotation is observable to the failover
|
|
99
|
+
# chaos probe (f05) and to operators.
|
|
98
100
|
@failed_over_to_secondary = false
|
|
99
101
|
end
|
|
100
102
|
|
|
@@ -269,7 +271,13 @@ module Quonfig
|
|
|
269
271
|
# gotcha and solve it the same way: per-chunk reset, async close on
|
|
270
272
|
# expiry (chaos scenario 02 — sse_silent_stall).
|
|
271
273
|
def stream_once(&block)
|
|
272
|
-
|
|
274
|
+
base = current_url
|
|
275
|
+
# f05 latch: record if this connection attempt targets a non-primary
|
|
276
|
+
# stream leg. With the pin in #current_url this is structurally false;
|
|
277
|
+
# measuring at the dial site (rather than inside URL selection) keeps a
|
|
278
|
+
# rotation regression observable to the chaos probe.
|
|
279
|
+
@failed_over_to_secondary = true if base != @prefab_options.sse_api_urls.first
|
|
280
|
+
url = "#{base}/api/v2/sse/config"
|
|
273
281
|
cursor = current_cursor
|
|
274
282
|
@logger.debug "SSE Streaming Connect to #{url} start_at #{cursor.inspect}"
|
|
275
283
|
|
|
@@ -405,14 +413,15 @@ module Quonfig
|
|
|
405
413
|
end
|
|
406
414
|
end
|
|
407
415
|
|
|
408
|
-
#
|
|
409
|
-
#
|
|
410
|
-
#
|
|
416
|
+
# The stream leg is PINNED to the primary stream URL (sse_api_urls[0])
|
|
417
|
+
# and retried forever with backoff — SSE never fails over (f05 invariant,
|
|
418
|
+
# qfg-41nh.6). Failover is HTTP-poll only: the config-fetch hedge keeps
|
|
419
|
+
# BOTH api_urls legs, but the live stream never repoints to
|
|
420
|
+
# stream.secondary (which is sized for poll rps, not SSE herds, and would
|
|
421
|
+
# silently bind freshness to the mirror). Matches sdk-go's sseClient — a
|
|
422
|
+
# single pinned URL with reconnect backoff (sse_client.go runLoop).
|
|
411
423
|
def current_url
|
|
412
|
-
|
|
413
|
-
@source_index = (@source_index + 1) % urls.size
|
|
414
|
-
@failed_over_to_secondary = true if @source_index.positive?
|
|
415
|
-
urls[@source_index]
|
|
424
|
+
@prefab_options.sse_api_urls.first
|
|
416
425
|
end
|
|
417
426
|
|
|
418
427
|
# Internal: HTTP-status sentinel error for non-200 SSE responses. Surfaces
|
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Quonfig
|
|
4
|
+
module Telemetry
|
|
5
|
+
# Accumulates failover-behavior counters over a flush window: how many times
|
|
6
|
+
# the config-fetch hedge fired its secondary leg, how many installs the
|
|
7
|
+
# reject-older ordering guard dropped, and which upstream leg served each
|
|
8
|
+
# successful HTTP install. Every counter is additive and carries NO user
|
|
9
|
+
# data.
|
|
10
|
+
#
|
|
11
|
+
# The aggregator is independently thread-safe (its own mutex) and is written
|
|
12
|
+
# directly from the failover call sites in ConfigLoader rather than through
|
|
13
|
+
# any queue — the call rate is per-config-refresh, not per-evaluation, so a
|
|
14
|
+
# plain mutex has negligible overhead.
|
|
15
|
+
#
|
|
16
|
+
# Wire shape matches sdk-go's FailoverEvent (the canonical reference) and
|
|
17
|
+
# api-telemetry's failover schema EXACTLY — camelCase keys, unix-millis
|
|
18
|
+
# window. #drain_event returns +nil+ when every counter is zero, so a healthy
|
|
19
|
+
# steady-state client emits no failover event at all.
|
|
20
|
+
#
|
|
21
|
+
# { "failover": {
|
|
22
|
+
# "start": <int ms>, "end": <int ms>,
|
|
23
|
+
# "hedgeFired": <int>, "guardRejected": <int>,
|
|
24
|
+
# "resolvedFromPrimary": <int>, "resolvedFromSecondary": <int>,
|
|
25
|
+
# "resolvedFromLkg": <int>
|
|
26
|
+
# } }
|
|
27
|
+
class FailoverAggregator
|
|
28
|
+
def initialize
|
|
29
|
+
@mutex = Mutex.new
|
|
30
|
+
@start_at_ms = nil
|
|
31
|
+
@hedge_fired = 0
|
|
32
|
+
@guard_rejected = 0
|
|
33
|
+
@resolved_from_primary = 0
|
|
34
|
+
@resolved_from_secondary = 0
|
|
35
|
+
# Reserved for a future last-known-good delivery path; backends emit 0.
|
|
36
|
+
@resolved_from_lkg = 0
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
# Count one config-fetch cycle whose hedge fired the secondary leg (the
|
|
40
|
+
# primary was slow past the hedge delay or errored fast).
|
|
41
|
+
def record_hedge_fired
|
|
42
|
+
@mutex.synchronize do
|
|
43
|
+
@start_at_ms ||= Quonfig::TimeHelpers.now_in_ms
|
|
44
|
+
@hedge_fired += 1
|
|
45
|
+
end
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
# Count one install dropped by the reject-older ordering guard (an
|
|
49
|
+
# equal-or-older snapshot on any install path — HTTP config-fetch or SSE).
|
|
50
|
+
def record_guard_rejected
|
|
51
|
+
@mutex.synchronize do
|
|
52
|
+
@start_at_ms ||= Quonfig::TimeHelpers.now_in_ms
|
|
53
|
+
@guard_rejected += 1
|
|
54
|
+
end
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
# Count one successful HTTP install by the leg that served it: source_index
|
|
58
|
+
# 0 is the primary, any index > 0 is a failover/secondary leg. A nil or
|
|
59
|
+
# negative index (SSE / datadir install with no HTTP leg) is ignored.
|
|
60
|
+
def record_resolved_from(source_index)
|
|
61
|
+
return if source_index.nil? || source_index.negative?
|
|
62
|
+
|
|
63
|
+
@mutex.synchronize do
|
|
64
|
+
@start_at_ms ||= Quonfig::TimeHelpers.now_in_ms
|
|
65
|
+
if source_index.zero?
|
|
66
|
+
@resolved_from_primary += 1
|
|
67
|
+
else
|
|
68
|
+
@resolved_from_secondary += 1
|
|
69
|
+
end
|
|
70
|
+
end
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
# Return the window's counters as a telemetry event hash and reset state.
|
|
74
|
+
# Returns +nil+ when no failover activity occurred (every counter zero).
|
|
75
|
+
def drain_event
|
|
76
|
+
@mutex.synchronize do
|
|
77
|
+
if @hedge_fired.zero? && @guard_rejected.zero? &&
|
|
78
|
+
@resolved_from_primary.zero? && @resolved_from_secondary.zero? &&
|
|
79
|
+
@resolved_from_lkg.zero?
|
|
80
|
+
return nil
|
|
81
|
+
end
|
|
82
|
+
|
|
83
|
+
event = {
|
|
84
|
+
'failover' => {
|
|
85
|
+
'start' => @start_at_ms || Quonfig::TimeHelpers.now_in_ms,
|
|
86
|
+
'end' => Quonfig::TimeHelpers.now_in_ms,
|
|
87
|
+
'hedgeFired' => @hedge_fired,
|
|
88
|
+
'guardRejected' => @guard_rejected,
|
|
89
|
+
'resolvedFromPrimary' => @resolved_from_primary,
|
|
90
|
+
'resolvedFromSecondary' => @resolved_from_secondary,
|
|
91
|
+
'resolvedFromLkg' => @resolved_from_lkg
|
|
92
|
+
}
|
|
93
|
+
}
|
|
94
|
+
|
|
95
|
+
reset
|
|
96
|
+
event
|
|
97
|
+
end
|
|
98
|
+
end
|
|
99
|
+
|
|
100
|
+
private
|
|
101
|
+
|
|
102
|
+
# Caller holds @mutex.
|
|
103
|
+
def reset
|
|
104
|
+
@start_at_ms = nil
|
|
105
|
+
@hedge_fired = 0
|
|
106
|
+
@guard_rejected = 0
|
|
107
|
+
@resolved_from_primary = 0
|
|
108
|
+
@resolved_from_secondary = 0
|
|
109
|
+
@resolved_from_lkg = 0
|
|
110
|
+
end
|
|
111
|
+
end
|
|
112
|
+
end
|
|
113
|
+
end
|
|
@@ -13,7 +13,8 @@ module Quonfig
|
|
|
13
13
|
# "events": [
|
|
14
14
|
# { "summaries": { "start": ..., "end": ..., "summaries": [...] } },
|
|
15
15
|
# { "contextShapes": { "shapes": [...] } },
|
|
16
|
-
# { "exampleContexts": { "examples": [...] } }
|
|
16
|
+
# { "exampleContexts": { "examples": [...] } },
|
|
17
|
+
# { "failover": { "start": ..., "end": ..., "hedgeFired": ..., ... } }
|
|
17
18
|
# ]
|
|
18
19
|
# }
|
|
19
20
|
#
|
|
@@ -30,6 +31,7 @@ module Quonfig
|
|
|
30
31
|
context_shape_aggregator: nil,
|
|
31
32
|
example_contexts_aggregator: nil,
|
|
32
33
|
evaluation_summaries_aggregator: nil,
|
|
34
|
+
failover_aggregator: nil,
|
|
33
35
|
sync_interval: nil,
|
|
34
36
|
http_connection: nil)
|
|
35
37
|
@options = options
|
|
@@ -39,6 +41,11 @@ module Quonfig
|
|
|
39
41
|
@context_shape_aggregator = context_shape_aggregator
|
|
40
42
|
@example_contexts_aggregator = example_contexts_aggregator
|
|
41
43
|
@evaluation_summaries_aggregator = evaluation_summaries_aggregator
|
|
44
|
+
# Failover counters carry no user data and are the operational signal for
|
|
45
|
+
# the secondary-delivery hardening (qfg-41nh.18), so they ride the
|
|
46
|
+
# existing telemetry stream. The ConfigLoader records directly into this
|
|
47
|
+
# aggregator at the failover call sites; the reporter only drains it.
|
|
48
|
+
@failover_aggregator = failover_aggregator
|
|
42
49
|
@http_connection = http_connection
|
|
43
50
|
@sync_interval = calculate_sync_interval(sync_interval)
|
|
44
51
|
@stopped = Concurrent::AtomicBoolean.new(false)
|
|
@@ -125,6 +132,9 @@ module Quonfig
|
|
|
125
132
|
if (example_event = @example_contexts_aggregator&.drain_event)
|
|
126
133
|
events << example_event
|
|
127
134
|
end
|
|
135
|
+
if (failover_event = @failover_aggregator&.drain_event)
|
|
136
|
+
events << failover_event
|
|
137
|
+
end
|
|
128
138
|
|
|
129
139
|
return if events.empty?
|
|
130
140
|
|
data/lib/quonfig/version.rb
CHANGED
data/lib/quonfig.rb
CHANGED
|
@@ -61,6 +61,7 @@ require 'quonfig/telemetry/context_shape'
|
|
|
61
61
|
require 'quonfig/telemetry/context_shape_aggregator'
|
|
62
62
|
require 'quonfig/telemetry/example_contexts_aggregator'
|
|
63
63
|
require 'quonfig/telemetry/evaluation_summaries_aggregator'
|
|
64
|
+
require 'quonfig/telemetry/failover_aggregator'
|
|
64
65
|
require 'quonfig/telemetry/telemetry_reporter'
|
|
65
66
|
require 'quonfig/client'
|
|
66
67
|
require 'quonfig/bound_client'
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: quonfig
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.2.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Jeff Dwyer
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-07-
|
|
11
|
+
date: 2026-07-08 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: activesupport
|
|
@@ -130,6 +130,7 @@ files:
|
|
|
130
130
|
- lib/quonfig/telemetry/context_shape_aggregator.rb
|
|
131
131
|
- lib/quonfig/telemetry/evaluation_summaries_aggregator.rb
|
|
132
132
|
- lib/quonfig/telemetry/example_contexts_aggregator.rb
|
|
133
|
+
- lib/quonfig/telemetry/failover_aggregator.rb
|
|
133
134
|
- lib/quonfig/telemetry/telemetry_reporter.rb
|
|
134
135
|
- lib/quonfig/time_helpers.rb
|
|
135
136
|
- lib/quonfig/types.rb
|