quonfig 1.3.0 → 1.4.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -21,14 +21,16 @@ module Quonfig
21
21
  LOG = Quonfig::InternalLogger.new(self)
22
22
 
23
23
  # qfg-ryov: instance registry for the Process._fork hook. Every live
24
- # Client is tracked here so the hook can fan out before_fork_in_parent /
25
- # after_fork_in_child across all of them without the customer needing to
26
- # name a specific instance. ObjectSpace::WeakMap means a Client that goes
27
- # out of scope is GC'd without leaking through this registry. Stopped
28
- # Clients stay in the registry until GC; both fork hooks early-return on
29
- # +@stopped+ so a stopped instance is effectively a no-op. (We don't use
30
- # WeakMap#delete because it was added in Ruby 3.3 and the matrix still
31
- # includes 3.2.)
24
+ # Client is tracked here so the hook can fan out after_fork_in_child
25
+ # across all of them without the customer needing to name a specific
26
+ # instance. ObjectSpace::WeakMap means a Client that goes out of scope is
27
+ # GC'd without leaking through this registry. Stopped Clients stay in the
28
+ # registry until GC; after_fork_in_child early-returns on +@stopped+ so a
29
+ # stopped instance is effectively a no-op. (We don't use WeakMap#delete
30
+ # because it was added in Ruby 3.3 and the matrix still includes 3.2.)
31
+ #
32
+ # The registry is read in the CHILD only (qfg-lv4n.1) — the hook does
33
+ # nothing on the parent side, so no lock is taken across the syscall.
32
34
  @instances = ObjectSpace::WeakMap.new
33
35
  @instances_mutex = Mutex.new
34
36
 
@@ -43,8 +45,7 @@ module Quonfig
43
45
  end
44
46
  end
45
47
 
46
- attr_reader :options, :resolver, :store, :evaluator, :instance_hash,
47
- :config_loader, :telemetry_reporter
48
+ attr_reader :options, :instance_hash, :telemetry_reporter
48
49
 
49
50
  def initialize(options = nil, store: nil, **option_kwargs)
50
51
  @options =
@@ -82,6 +83,25 @@ module Quonfig
82
83
  @sse_ever_connected = false
83
84
  @fallback_engage_timer = nil
84
85
  @sse_terminal_failure = false
86
+ # The process that owns this client's threads, sockets, and store.
87
+ # Re-stamped when a child takes ownership in #rebuild_in_child!. A pid
88
+ # mismatch is PROOF that we are looking at a fork(2) child, which is
89
+ # what makes #owned_by_this_process? exact. See that method for why
90
+ # thread liveness could not answer the question.
91
+ @owner_pid = Process.pid
92
+ # Post-fork lazy re-initialization (qfg-lv4n.1). Set by
93
+ # +after_fork_in_child+; cleared by the first use of the client in the
94
+ # child. See #ensure_initialized_after_fork.
95
+ @fork_rebuild_pending = false
96
+ @fork_rebuild_mutex = Mutex.new
97
+ # The thread currently running the rebuild, so a re-entrant read (a
98
+ # SemanticLoggerFilter or stdlib formatter that calls +get+ from inside
99
+ # the rebuild's own logging) does not deadlock on the non-reentrant
100
+ # Mutex above.
101
+ @fork_rebuild_owner = nil
102
+ # Sticky init error under +on_init_failure: :raise+ (see
103
+ # #raise_sticky_fork_init_error).
104
+ @fork_rebuild_error = nil
85
105
 
86
106
  # If the caller injected a store, we're in test/bootstrap mode; skip I/O.
87
107
  return if store
@@ -103,6 +123,7 @@ module Quonfig
103
123
  # ---- Lookup --------------------------------------------------------
104
124
 
105
125
  def get(key, default = NO_DEFAULT_PROVIDED, jit_context = NO_DEFAULT_PROVIDED)
126
+ ensure_initialized_after_fork
106
127
  ctx = build_context(jit_context)
107
128
  record_context_for_telemetry(ctx)
108
129
  result =
@@ -186,10 +207,12 @@ module Quonfig
186
207
  end
187
208
 
188
209
  def defined?(key)
210
+ ensure_initialized_after_fork
189
211
  !@store.get(key).nil?
190
212
  end
191
213
 
192
214
  def keys
215
+ ensure_initialized_after_fork
193
216
  @store.keys
194
217
  end
195
218
 
@@ -312,51 +335,129 @@ module Quonfig
312
335
  end
313
336
 
314
337
  def stop
338
+ # Order matters (qfg-lv4n.1 D5). The flag goes up BEFORE we queue for
339
+ # the rebuild lock, so a post-fork rebuild already in flight sees it and
340
+ # skips starting an update channel and a telemetry reporter at all —
341
+ # otherwise it builds an SSE worker after we have finished tearing down
342
+ # and nothing is left holding a reference to close it.
315
343
  @stopped = true
316
- tear_down_threaded_components!
344
+ # A child that never used the client must be able to stop it without
345
+ # paying for a re-initialization it never asked for.
346
+ @fork_rebuild_pending = false
347
+ @fork_rebuild_error = nil
348
+
349
+ # ...and the teardown itself is serialized against the rebuild, so it
350
+ # can never interleave with component construction. Re-entrancy: if the
351
+ # rebuild is what called `stop` (a customer on_update/logger hook), this
352
+ # thread already holds the lock.
353
+ if @fork_rebuild_owner == Thread.current
354
+ tear_down_threaded_components!
355
+ else
356
+ @fork_rebuild_mutex.synchronize { tear_down_threaded_components! }
357
+ end
317
358
  end
318
359
 
319
- # qfg-ryov: pre-fork hook. Close the SSE worker, polling supervisor,
320
- # telemetry reporter, and any fallback-engage timer. Idempotent calling
321
- # twice is safe. Does NOT set @stopped: the client is still expected to
322
- # be usable post-fork via after_fork_in_child.
360
+ # @deprecated Since 1.4.0 the +Process._fork+ hook NO LONGER CALLS THIS.
361
+ # A fork must not disturb the process that forked: the parent keeps its
362
+ # SSE stream, its poller, and its telemetry reporter, and keeps serving
363
+ # live config (qfg-lv4n.1). This method is retained for semver and for
364
+ # the Ruby 3.0 manual-wiring path, where a customer who genuinely wants
365
+ # the parent torn down before a fork can still call it. Prefer +stop+
366
+ # if you want the client dead.
323
367
  #
324
- # Why this matters: Ruby threads do not survive fork(2). If we let the
325
- # child inherit a live Net::HTTP socket, both processes read from the
326
- # same fd and corrupt each other's bytes. Closing in the parent before
327
- # fork is the only safe shape.
368
+ # Closes the SSE worker, polling supervisor, telemetry reporter, datadir
369
+ # watcher, and any fallback-engage timer. Idempotent. Does NOT set
370
+ # +@stopped+, so +after_fork_in_child+ can still rebuild.
328
371
  def before_fork_in_parent
329
372
  return if @stopped
330
373
 
331
374
  tear_down_threaded_components!
332
375
  end
333
376
 
334
- # qfg-ryov: post-fork (in child) hook. Re-establish whatever threaded
335
- # components the client had pre-fork. No-op if the client was already
336
- # stopped (the customer asked for it to be dead do not resurrect),
337
- # or if the client is in datadir mode (no threaded components to start).
377
+ # Post-fork hook, run IN THE CHILD ONLY (see Quonfig::ForkSafety).
378
+ #
379
+ # Ruby threads do not survive fork(2), so everything threaded the child
380
+ # inherited is a dead reference. The child drops those references and
381
+ # rebuilds from scratch — matching Reforge's +Reforge.fork+, which simply
382
+ # constructs a brand-new client and lets the inherited one be collected.
383
+ #
384
+ # Two things we deliberately do NOT do to the inherited objects:
385
+ #
386
+ # * **Never close the inherited SSE socket.** fork(2) duplicates the fd,
387
+ # so the child's copy points at the connection the PARENT is still
388
+ # streaming on. Closing a TLS socket writes a +close_notify+ alert onto
389
+ # that shared connection and kills the parent's stream. Dropping the
390
+ # reference leaves the parent's fd untouched.
391
+ # * **Never join an inherited thread.** The thread does not exist in the
392
+ # child, so a +join+/+stop+ that waits on it blocks forever (see
393
+ # LaunchDarkly ruby-server-sdk PR #430: "close blocks forever, because
394
+ # EventProcessor#stop waits for a dispatcher thread that does not
395
+ # exist").
396
+ #
397
+ # No-op if the client was already stopped — the customer asked for it to
398
+ # be dead, and a fork must not resurrect it.
399
+ #
400
+ # Also a no-op in the process that OWNS the client, i.e. the PARENT —
401
+ # decided by a pid stamp, not by whether anything looks alive. Releases
402
+ # 1.0-1.3 documented calling this in the parent as the workaround for the
403
+ # parent going dark after a fork; on 1.4.0+ such a call would orphan the
404
+ # parent's live components and zero its store, so it is ignored (one
405
+ # debug line). See #owned_by_this_process?.
406
+ #
407
+ # The hook does NO I/O: no fetch, no socket, no thread. It throws away
408
+ # everything the child inherited — including the parent's config snapshot
409
+ # — and arms a flag. The child re-initializes on its FIRST use of the
410
+ # client (see #ensure_initialized_after_fork), exactly like a newly
411
+ # constructed client would. A child that never uses the client, which is
412
+ # most of them in a `Parallel.map` batch, costs nothing at all.
338
413
  def after_fork_in_child
339
414
  return if @stopped
415
+ return if owned_by_this_process?
340
416
 
341
- if @options.datadir
342
- start_datadir_watcher if @options.data_dir_auto_reload
343
- return
344
- end
345
-
346
- return if @config_loader.nil? # never finished network init (e.g. invalid key)
417
+ # The inherited Mutexes may be held by threads that no longer exist.
418
+ # Only this thread exists in a fresh child, so swapping them is safe.
419
+ @state_mutex = Mutex.new
420
+ @fork_rebuild_mutex = Mutex.new
421
+ @fork_rebuild_owner = nil
422
+ @fork_rebuild_error = nil
423
+ drop_inherited_threaded_components!
347
424
 
348
- # SSE state machine carries flags that no longer apply in the child
349
- # (the parent had connected, the parent had errored, etc.). Reset.
350
- @state_mutex.synchronize do
351
- @sse_state = :idle
352
- @sse_ever_connected = false
353
- @sse_terminal_failure = false
354
- end
425
+ # SSE state machine carries flags that describe the PARENT's session
426
+ # (it had connected, it had errored, ...). None of them apply here.
427
+ @sse_state = :idle
428
+ @sse_ever_connected = false
429
+ @sse_terminal_failure = false
430
+ @sse_error_callback = nil
431
+
432
+ # A client that never finished network init has nothing to rebuild.
433
+ return if @config_loader.nil? && !@options.datadir
434
+
435
+ # A brand-new, EMPTY store. The child must not evaluate from whatever
436
+ # snapshot the parent happened to hold at the instant of the fork: it
437
+ # fetches (or loads) its own on first use.
438
+ reset_store_in_child!
439
+
440
+ # A fresh instance hash (qfg-xcym). `@instance_hash` identifies this
441
+ # SDK instance in every telemetry payload, and app-quonfig's Debugger
442
+ # groups SDK last-seen by it — so a child that keeps the parent's hash
443
+ # collapses an 8-worker Puma cluster into ONE row with the parent's and
444
+ # the children's windows interleaved. Reforge does not have this
445
+ # problem: `Reforge.fork` builds a whole new Client, which mints its
446
+ # own. This MUST run before +rebuild_aggregators_in_child!+: the
447
+ # reporter captures the hash at construction, so minting after the
448
+ # rebuild would leave the child POSTing under the parent's identity.
449
+ # The parent's hash is untouched — the hook never runs there.
450
+ @instance_hash = SecureRandom.uuid
355
451
 
356
- sse_started = @options.enable_sse && start_sse
357
- start_polling if @options.fallback_poll_enabled && !sse_started
452
+ # Fresh aggregators. The parent flushes its own copy; a child that
453
+ # flushed inherited data would double-report it. The reporter is BUILT
454
+ # here (so the child's config loader points at the child's failover
455
+ # aggregator) but NOT started — starting it is I/O, and that waits for
456
+ # first use.
457
+ rebuild_aggregators_in_child!
358
458
 
359
- restart_telemetry_in_child
459
+ @forked_in_pid = Process.pid
460
+ @fork_rebuild_pending = true
360
461
  end
361
462
 
362
463
  # quonfig_sdk_worker_restart_total counter (Tier 1 supervisor contract).
@@ -409,7 +510,17 @@ module Quonfig
409
510
  def connection_state
410
511
  @state_mutex.synchronize do
411
512
  next :disconnected if @stopped
513
+ # Forked, not yet used: nothing has been fetched and nothing is
514
+ # running. Saying so is the honest answer, and a diagnostic must not
515
+ # be what triggers a blocking fetch.
516
+ next :initializing if @fork_rebuild_pending
412
517
  next :falling_back if @poll_supervisor&.alive?
518
+ # Liveness beats the stored flag (qfg-lv4n.1). A client whose SSE
519
+ # session was torn down keeps a stale @sse_state; answering
520
+ # :connected off that flag is how a dark client reported healthy for
521
+ # 13 days. If this client is supposed to have a live SSE worker and
522
+ # does not, it is disconnected — whatever the flag says.
523
+ next :disconnected if sse_channel_expected? && !sse_worker_alive?
413
524
  next :connected if @sse_state == :connected
414
525
  next :disconnected if @sse_state == :error
415
526
 
@@ -461,7 +572,65 @@ module Quonfig
461
572
  sse.failed_over_to_secondary?
462
573
  end
463
574
 
575
+ # ---- Component readers ---------------------------------------------
576
+ #
577
+ # Public since 1.0 and kept public for semver. Each one routes through
578
+ # the post-fork chokepoint: in a forked child that has not been used yet
579
+ # the raw components read an EMPTY store (+store.get+ answered nil,
580
+ # +resolver.get+ raised MissingDefaultError), so handing them back
581
+ # without re-initializing is handing back a component that lies
582
+ # (qfg-lv4n.1 D6).
583
+
584
+ # @return [Quonfig::ConfigStore] the store backing this client.
585
+ # @note In a forked child, reading this triggers the lazy post-fork
586
+ # re-initialization (see #after_fork_in_child) — it can block on the
587
+ # child's own config fetch.
588
+ def store
589
+ ensure_initialized_after_fork
590
+ @store
591
+ end
592
+
593
+ # @return [Quonfig::Resolver]
594
+ # @note (see #store)
595
+ def resolver
596
+ ensure_initialized_after_fork
597
+ @resolver
598
+ end
599
+
600
+ # @return [Quonfig::Evaluator]
601
+ # @note (see #store)
602
+ def evaluator
603
+ ensure_initialized_after_fork
604
+ @evaluator
605
+ end
606
+
607
+ # @return [Quonfig::ConfigLoader, nil] nil in datadir mode.
608
+ # @note (see #store)
609
+ def config_loader
610
+ ensure_initialized_after_fork
611
+ @config_loader
612
+ end
613
+
614
+ # A client to use in a forked child.
615
+ #
616
+ # On Ruby 3.1+ the +Process._fork+ hook has already prepared THIS client
617
+ # in the child by the time any user code runs there (inherited threads
618
+ # and store dropped, re-initialization armed for first use). A call here
619
+ # in that child — the +on_worker_boot { Quonfig.fork }+ line the 1.0–1.3
620
+ # README taught — therefore returns +self+: the hook already did what the
621
+ # caller is asking for. Building a second client instead would discard
622
+ # the prepared one and pay an eager second fetch, or, after first use,
623
+ # leave the worker holding two live SSE streams and two reporters with
624
+ # the first pair orphaned where +stop+ can never reach it (qfg-4t5o).
625
+ #
626
+ # Everywhere else — the owning process, Ruby 3.0 where there is no hook,
627
+ # a client that was +stop+ped before the fork — this builds a fresh
628
+ # client, which is the Ruby 3.0 manual-wiring path. The old client is
629
+ # never stopped: on 3.0 it is the inherited one, and stopping it would
630
+ # close the inherited socket and tear down the PARENT's stream.
464
631
  def fork
632
+ return self if @forked_in_pid == Process.pid
633
+
465
634
  self.class.new(@options.for_fork)
466
635
  end
467
636
 
@@ -471,10 +640,73 @@ module Quonfig
471
640
 
472
641
  private
473
642
 
474
- # Close every threaded component and drop its reference. Used by both
475
- # +stop+ (where @stopped is also flipped) and +before_fork_in_parent+
476
- # (where @stopped is left alone so the child can restart).
643
+ # True when THIS process is the one that owns the client — i.e. we are
644
+ # the parent, not a fork(2) child.
645
+ #
646
+ # The answer is a pid comparison against the stamp taken when the client
647
+ # was constructed (and re-taken when a child rebuilds it). A pid mismatch
648
+ # is PROOF of a fork child; a match is proof that nobody forked.
649
+ #
650
+ # It deliberately does NOT ask whether any component looks alive, which
651
+ # is what 1.4.0 shipped and what got this wrong at both ends:
652
+ #
653
+ # * **False negative in a real child.** +on_update+ runs on the SSE worker
654
+ # thread, so a customer who forks from that callback forks ON it — and
655
+ # the inherited +@worker.alive?+ is therefore true in the child. The
656
+ # child was classified as the parent, ignored the hook, served the
657
+ # parent's snapshot forever and reported +:connected+ (qfg-lv4n.1 E1).
658
+ # * **False positive in a real parent.** A datadir client with
659
+ # auto-reload off and no SDK key has no threads and no reporter at all,
660
+ # so the guard let a parent-side call through and it wiped the live
661
+ # store (qfg-lv4n.1 E4/D4).
662
+ #
663
+ # The parent case is what makes a stray +after_fork_in_child+ call a
664
+ # no-op. Releases 1.0-1.3 documented exactly that call as the workaround
665
+ # for the parent-keeps-evaluating topology, and that code is still out
666
+ # there: on 1.4.0+ it would orphan the live SSE worker and its stream,
667
+ # zero the store, and stop the owner's telemetry reporter.
668
+ def owned_by_this_process?
669
+ return false unless @owner_pid == Process.pid
670
+
671
+ LOG.debug '[quonfig] after_fork_in_child called in the process that OWNS the client ' \
672
+ "(pid=#{Process.pid}); ignoring. Since 1.4.0 a fork never touches the " \
673
+ 'process that forked, and the child-side rebuild is automatic on Ruby 3.1+.'
674
+ true
675
+ end
676
+
677
+ # True when this client is a network-mode client that asked for SSE, i.e.
678
+ # one that is SUPPOSED to be holding a live stream. Datadir clients and
679
+ # store-injected (test/bootstrap) clients never are, so their
680
+ # +connection_state+ keeps deriving from envelope installs alone.
681
+ def sse_channel_expected?
682
+ return false if @options.datadir
683
+ return false unless @options.enable_sse
684
+
685
+ !@config_loader.nil?
686
+ end
687
+
688
+ # Is there an SSE worker thread actually running right now? Note this
689
+ # stays true across a reconnect: the worker owns the retry loop, so a
690
+ # blip does not read as "no channel".
691
+ def sse_worker_alive?
692
+ sse = @sse_client
693
+ return false if sse.nil?
694
+ return true unless sse.respond_to?(:alive?)
695
+
696
+ sse.alive?
697
+ end
698
+
699
+ # Close every threaded component and drop its reference. Used by +stop+
700
+ # (where @stopped is also flipped) and by the deprecated manual
701
+ # +before_fork_in_parent+ (where @stopped is left alone). NOT reachable
702
+ # from the fork hook any more — a fork never touches the process that
703
+ # forked (qfg-lv4n.1).
477
704
  def tear_down_threaded_components!
705
+ # The SSE state machine describes a session that no longer exists.
706
+ # Leaving @sse_state == :connected behind is how `connection_state`
707
+ # came to answer :connected for a client with nothing alive.
708
+ @state_mutex.synchronize { @sse_state = :idle }
709
+
478
710
  begin
479
711
  @sse_client&.close
480
712
  rescue StandardError => e
@@ -506,11 +738,233 @@ module Quonfig
506
738
  @datadir_watcher = nil
507
739
  end
508
740
 
509
- # Rebuild the telemetry reporter in the child after fork. Mirrors the
510
- # original initialize_telemetry path fresh aggregators, fresh reporter.
511
- def restart_telemetry_in_child
741
+ # Drop every inherited threaded component WITHOUT closing, stopping, or
742
+ # joining it. See the comment on +after_fork_in_child+ for why touching
743
+ # these objects in the child is actively harmful (shared socket fds,
744
+ # threads that do not exist). Reforge, LaunchDarkly, dd-trace-rb,
745
+ # redis-client and connection_pool all do exactly this.
746
+ def drop_inherited_threaded_components!
747
+ inherited_reporter = @telemetry_reporter
748
+
749
+ @sse_client = nil
750
+ @poll_supervisor = nil
512
751
  @telemetry_reporter = nil
513
- initialize_telemetry
752
+ @datadir_watcher = nil
753
+ @fallback_engage_timer = nil
754
+
755
+ # Dropping our reference is not enough for the reporter: its
756
+ # `Kernel.at_exit { final_drain_on_exit }` closure is process-wide, it
757
+ # was copied by fork(2), and it still holds a full copy of the PARENT's
758
+ # un-flushed telemetry window. The reporter's own owner-pid guard is
759
+ # what makes that closure inert (see TelemetryReporter#start); this
760
+ # additionally makes the copied window unreachable. Neither stops,
761
+ # closes, nor joins anything.
762
+ begin
763
+ inherited_reporter&.discard_inherited!
764
+ rescue StandardError => e
765
+ LOG.debug "Error discarding inherited telemetry reporter: #{e.message}"
766
+ end
767
+ end
768
+
769
+ # Lazy post-fork re-initialization. Called from every read entry point
770
+ # (+get+, +evaluate_details+, +defined?+, +keys+) — the flag read is a
771
+ # plain boolean, so the steady-state cost is one comparison per lookup.
772
+ #
773
+ # The first caller in the child does what +Client.new+ does: its own
774
+ # config fetch under the configured init timeout and +on_init_failure+
775
+ # policy, then its own SSE stream (or fallback poller) and its own
776
+ # telemetry reporter. It BLOCKS, so that first lookup already reflects
777
+ # the child's own current config.
778
+ #
779
+ # +connection_state+ deliberately does NOT trigger this: a diagnostic
780
+ # must never open a socket. It reports +:initializing+ while a rebuild is
781
+ # pending, which is exactly what the client is.
782
+ def ensure_initialized_after_fork
783
+ # Hot path: two ivar reads and no lock. Both are falsy for every client
784
+ # that has never been through a fork.
785
+ return unless @fork_rebuild_pending || @fork_rebuild_error
786
+ # Re-entrancy guard: a customer logger (SemanticLoggerFilter, stdlib
787
+ # formatter) that evaluates a config from inside the rebuild would
788
+ # otherwise deadlock on the non-reentrant Mutex. Such a call sees the
789
+ # half-built client, which is the same thing Client.new gives a logger
790
+ # that fires during construction.
791
+ return if @fork_rebuild_owner == Thread.current
792
+
793
+ run_pending_child_rebuild if @fork_rebuild_pending
794
+ raise_sticky_fork_init_error if @fork_rebuild_error
795
+ end
796
+
797
+ # Run the rebuild under the lock, or block until whoever is running it is
798
+ # done. The flag stays TRUE for the whole rebuild, which is what makes
799
+ # every other first-use caller take the mutex and WAIT rather than sail
800
+ # past on the unlocked fast path and evaluate against the empty store.
801
+ def run_pending_child_rebuild
802
+ @fork_rebuild_mutex.synchronize do
803
+ # Lost the race: the winner already rebuilt (or `stop` disarmed us).
804
+ return unless @fork_rebuild_pending
805
+ return if @stopped
806
+
807
+ @fork_rebuild_owner = Thread.current
808
+ begin
809
+ rebuild_in_child!
810
+ rescue StandardError => e
811
+ # Handled: the child gets whatever healing path its mode allows, so
812
+ # the next lookup must not re-run the blocking fetch. (The datadir
813
+ # recovery path may deliberately re-arm — see
814
+ # #recover_datadir_child_after_failed_rebuild.)
815
+ @fork_rebuild_pending = false
816
+ handle_child_rebuild_failure(e)
817
+ ensure
818
+ @fork_rebuild_owner = nil
819
+ # #rebuild_in_child! disarms the flag itself the moment the child
820
+ # has a live path to config. Anything that escapes before that —
821
+ # including a non-StandardError such as rack-timeout's
822
+ # RequestTimeoutException, Ruby 3.3's Timeout::ExitException, or a
823
+ # Thread#kill, none of which the rescue above can see — leaves the
824
+ # flag armed so the NEXT call retries instead of leaving the child
825
+ # dark forever (qfg-lv4n.1 D2).
826
+ @fork_rebuild_pending = false if @stopped
827
+ end
828
+ end
829
+ end
830
+
831
+ # Under +on_init_failure: :raise+ a failed rebuild raises out of the call
832
+ # that triggered it, exactly as +Client.new+ would — and keeps raising on
833
+ # subsequent calls (without re-fetching) until the update channel heals
834
+ # the store. Under +:return+ nothing is stored here and this never fires.
835
+ def raise_sticky_fork_init_error
836
+ err = @fork_rebuild_error
837
+ return if err.nil?
838
+
839
+ if ready?
840
+ # The SSE stream (or the poller) installed an envelope: the client is
841
+ # serving real config again, so the init failure is history.
842
+ @fork_rebuild_error = nil
843
+ return
844
+ end
845
+
846
+ raise err
847
+ end
848
+
849
+ # A rebuild that raised. Log it, give the child whatever healing path its
850
+ # mode has, and honor +on_init_failure+.
851
+ def handle_child_rebuild_failure(err)
852
+ LOG.error "[quonfig] post-fork re-initialization failed: #{err.class}: #{err.message}"
853
+
854
+ if @options.datadir
855
+ recover_datadir_child_after_failed_rebuild
856
+ else
857
+ begin
858
+ start_update_channel if @sse_client.nil? && @poll_supervisor.nil?
859
+ rescue StandardError => e
860
+ LOG.error "[quonfig] post-fork update channel failed to start: #{e.class}: #{e.message}"
861
+ end
862
+ end
863
+
864
+ return unless @options.on_init_failure == Quonfig::Options::ON_INITIALIZATION_FAILURE::RAISE
865
+
866
+ # Parity with a fresh Client.new, which raises under :raise. Stored so
867
+ # later calls keep raising rather than re-running the fetch on every
868
+ # lookup.
869
+ @fork_rebuild_error = err
870
+ raise err
871
+ end
872
+
873
+ # A datadir client is configured OFFLINE: it has no config loader, so
874
+ # opening an SSE stream here dials the network on a customer who asked
875
+ # for none and then blows up on every envelope that arrives
876
+ # ("undefined method `apply_envelope' for nil"). The healing path for a
877
+ # datadir child is the filesystem: start the watcher if auto-reload is on
878
+ # so a repaired workspace is picked up, and otherwise re-arm the rebuild
879
+ # so the next use retries the load (qfg-lv4n.1 D3).
880
+ def recover_datadir_child_after_failed_rebuild
881
+ begin
882
+ start_datadir_watcher if @options.data_dir_auto_reload && @datadir_watcher.nil?
883
+ rescue StandardError => e
884
+ LOG.error "[quonfig] post-fork datadir watcher failed to start: #{e.class}: #{e.message}"
885
+ end
886
+
887
+ return unless @datadir_watcher.nil?
888
+
889
+ @fork_rebuild_pending = true
890
+ end
891
+
892
+ # The child's own re-initialization, run on first use. Mirrors what
893
+ # +Client.new+ does for this client's mode, and logs one info line so a
894
+ # customer grepping their logs can see the SDK noticed the fork.
895
+ def rebuild_in_child!
896
+ # This process is taking ownership of the client. Re-stamping here is
897
+ # what keeps #owned_by_this_process? exact for everything that follows:
898
+ # a manual +after_fork_in_child+ in THIS child is now correctly a
899
+ # no-op, and a grandchild forked from here is still detected by pid.
900
+ @owner_pid = Process.pid
901
+ components = []
902
+
903
+ if @options.datadir
904
+ load_datadir_into_store
905
+ components << 'datadir'
906
+ start_datadir_watcher if @options.data_dir_auto_reload
907
+ components << 'datadir-watcher' if @datadir_watcher
908
+ else
909
+ initialize_network_mode
910
+ components << 'config' if ready?
911
+ components << 'sse' if @sse_client
912
+ components << 'polling' if @poll_supervisor
913
+ end
914
+
915
+ # The child now has a live path to config (a stream/poller, or a loaded
916
+ # datadir). Disarm HERE, not in the caller: everything above is
917
+ # retryable and must stay armed if it is interrupted, and everything
918
+ # below must never re-run the fetch or dial a second stream.
919
+ @fork_rebuild_pending = false
920
+
921
+ unless @stopped
922
+ @telemetry_reporter&.start
923
+ components << 'telemetry' if @telemetry_reporter
924
+ end
925
+
926
+ log_child_rebuild(components)
927
+ end
928
+
929
+ # A brand-new store (plus the evaluator, resolver, and config loader that
930
+ # read it) so the child starts from nothing and installs its own envelope.
931
+ # Two things this buys beyond "no stale config": the child's first
932
+ # envelope is ACCEPTED rather than dropped by the reject-older guard as
933
+ # same-generation, and a fork that lands mid-install can no longer hand
934
+ # the child a half-written store.
935
+ def reset_store_in_child!
936
+ @store = Quonfig::ConfigStore.new
937
+ @evaluator = Quonfig::Evaluator.new(@store, env_id: @options.environment)
938
+ @resolver = Quonfig::Resolver.new(@store, @evaluator)
939
+ @last_successful_refresh = nil
940
+ return if @options.datadir
941
+
942
+ @config_loader = Quonfig::ConfigLoader.new(@store, @options, failover_aggregator: @failover_aggregator)
943
+ end
944
+
945
+ # Replace every aggregator with a fresh, empty one so the child never
946
+ # re-reports data the parent collected (and is still going to flush from
947
+ # its own copy). Mirrors what a brand-new Client.new would allocate.
948
+ def rebuild_aggregators_in_child!
949
+ @failover_aggregator = Quonfig::Telemetry::FailoverAggregator.new
950
+ # The ConfigLoader records hedge/guard/resolved-from at its failover
951
+ # call sites, so it has to point at the child's aggregator too — the
952
+ # inherited one is now the parent's private object.
953
+ @config_loader.failover_aggregator = @failover_aggregator if @config_loader.respond_to?(:failover_aggregator=)
954
+
955
+ # initialize_telemetry allocates fresh context/example/summaries
956
+ # aggregators and a fresh reporter. It is NOT started here: starting the
957
+ # reporter is I/O and a thread, and both wait for the child's first use.
958
+ @telemetry_reporter = nil
959
+ initialize_telemetry(start: false)
960
+ end
961
+
962
+ # One line, at info, so a customer can see in their logs that the SDK
963
+ # noticed the fork and rebuilt. Deliberately not a warning: forking is
964
+ # normal and expected.
965
+ def log_child_rebuild(components)
966
+ list = components.empty? ? 'none' : components.join(',')
967
+ LOG.info "[quonfig] re-initialized after fork pid=#{Process.pid} components=#{list}"
514
968
  end
515
969
 
516
970
  # Stamp +last_successful_refresh+ at install time. Called by every code
@@ -668,7 +1122,7 @@ module Quonfig
668
1122
  # Construct and start the telemetry reporter if the options permit it.
669
1123
  # The reporter runs on a background thread and periodically POSTs
670
1124
  # context-shape and example-context batches to +telemetry_destination+.
671
- def initialize_telemetry
1125
+ def initialize_telemetry(start: true)
672
1126
  shape_aggregator = nil
673
1127
  example_aggregator = nil
674
1128
  summaries_aggregator = nil
@@ -704,6 +1158,7 @@ module Quonfig
704
1158
  )
705
1159
 
706
1160
  return unless @telemetry_reporter.enabled?
1161
+ return unless start
707
1162
 
708
1163
  @telemetry_reporter.start
709
1164
  rescue StandardError => e
@@ -826,15 +1281,32 @@ module Quonfig
826
1281
  warn_if_hedge_abort_exceeds_init_timeout
827
1282
  warn_if_explicit_api_urls_disables_failover
828
1283
 
829
- @config_loader = Quonfig::ConfigLoader.new(@store, @options, failover_aggregator: @failover_aggregator)
1284
+ # ||=: after a fork the child already built its loader over its fresh
1285
+ # store (see #reset_store_in_child!).
1286
+ @config_loader ||= Quonfig::ConfigLoader.new(@store, @options, failover_aggregator: @failover_aggregator)
830
1287
 
831
1288
  perform_initial_fetch
1289
+ start_update_channel
1290
+ end
832
1291
 
833
- sse_started = @options.enable_sse && start_sse
1292
+ # SSE if enabled and it comes up; otherwise the HTTP polling fallback.
1293
+ # Polling is a fallback: if SSE is off or failed to start, poll. This
1294
+ # avoids double-work when SSE is healthy but still refreshes the store in
1295
+ # environments that block SSE (corporate proxies, Lambda, etc.).
1296
+ def start_update_channel
1297
+ # A `stop` that raced the post-fork rebuild must win: never dial a
1298
+ # stream for a client the customer has already killed (qfg-lv4n.1).
1299
+ return if @stopped
1300
+ # Idempotent. `rebuild_in_child!` disarms @fork_rebuild_pending only
1301
+ # AFTER initialize_network_mode has already started the channel, so a
1302
+ # non-StandardError landing in that window (rack-timeout,
1303
+ # Timeout::ExitException, Thread#kill) leaves the flag armed WITH a
1304
+ # live stream. The retry re-runs initialize_network_mode; without this
1305
+ # guard it dialled a SECOND stream and overwrote @sse_client, orphaning
1306
+ # the first worker where `stop` could never reach it (qfg-lv4n.1 E2).
1307
+ return if sse_worker_alive? || @poll_supervisor&.alive?
834
1308
 
835
- # Polling is a fallback: if SSE is off or failed to start, poll. This
836
- # avoids double-work when SSE is healthy but still refreshes the store
837
- # in environments that block SSE (corporate proxies, Lambda, etc.).
1309
+ sse_started = @options.enable_sse && start_sse
838
1310
  start_polling if @options.enable_polling && !sse_started
839
1311
  end
840
1312
 
@@ -959,6 +1431,7 @@ module Quonfig
959
1431
  # Returns true if SSE started successfully, false otherwise. A false here
960
1432
  # signals the caller to fall back to polling.
961
1433
  def start_sse
1434
+ return false if @stopped
962
1435
  return false if @options.sse_api_urls.nil? || @options.sse_api_urls.empty?
963
1436
 
964
1437
  @sse_client = Quonfig::SSEConfigClient.new(
@@ -1143,6 +1616,7 @@ module Quonfig
1143
1616
  # caller's context, after coercing/checking +expected_type+. Never
1144
1617
  # raises; all exceptions become ERROR details.
1145
1618
  def evaluate_details(key, expected_type, context)
1619
+ ensure_initialized_after_fork
1146
1620
  jit = context == NO_DEFAULT_PROVIDED ? nil : context
1147
1621
  ctx = build_context(jit)
1148
1622
  record_context_for_telemetry(ctx)
@@ -1282,32 +1756,52 @@ module Quonfig
1282
1756
  end
1283
1757
  end
1284
1758
 
1285
- # qfg-ryov: hook into Process._fork so customers using Puma's clustered
1286
- # mode (or any preload/fork-worker server) don't have to wire
1287
- # +before_fork+/+on_worker_boot+ manually. Ruby 3.1+ routes every
1288
- # +Kernel#fork+/+Process.fork+ call through +Process._fork+, so a single
1289
- # prepend covers them all.
1759
+ # qfg-ryov / qfg-lv4n.1: hook into Process._fork so customers using Puma's
1760
+ # clustered mode (or any preload/fork-worker server, or a gem that forks
1761
+ # inside a job) don't have to wire +before_fork+/+on_worker_boot+ manually.
1762
+ # Ruby 3.1+ routes every +Kernel#fork+/+Process.fork+ call through
1763
+ # +Process._fork+, so a single prepend covers them all.
1290
1764
  #
1291
1765
  # Process._fork's contract:
1292
1766
  # - Called in the parent process before the fork syscall.
1293
1767
  # - Returns 0 in the child, child's pid in the parent.
1294
1768
  # - +super+ performs the actual fork.
1295
1769
  #
1296
- # The parent's view: SSE/polling/telemetry threads are torn down before
1297
- # the syscall so the child does not inherit a live Net::HTTP socket fd
1298
- # (which would corrupt both sides). The parent does NOT auto-restart
1299
- # that mirrors the Puma master use case where the master process no
1300
- # longer serves requests after spawning workers.
1770
+ # **The hook is child-only.** Nothing happens in the parent — not before
1771
+ # the syscall, not after it. A fork is somebody else's business; the
1772
+ # process that forked keeps its SSE stream, its poller, its telemetry
1773
+ # reporter, and its live config. This is Reforge's model (+Reforge.fork+
1774
+ # builds a new client in the child and never touches the old one) and
1775
+ # matches dd-trace-rb, redis-client and connection_pool, which all branch
1776
+ # on the child stage of +_fork+ only.
1777
+ #
1778
+ # It replaces the qfg-ryov shape, which tore the parent down before the
1779
+ # syscall on the theory that the child must not inherit a live socket fd.
1780
+ # That was wrong twice over: the child never touches the inherited fd (it
1781
+ # drops the reference — see Client#after_fork_in_child), and a Sidekiq
1782
+ # parent that forks a worker and keeps evaluating went dark for 13 days in
1783
+ # production.
1301
1784
  module ForkSafety
1302
1785
  def _fork
1303
- Quonfig::Client.each_instance(&:before_fork_in_parent)
1304
1786
  pid = super
1305
- Quonfig::Client.each_instance(&:after_fork_in_child) if pid.zero?
1787
+ if pid.zero?
1788
+ # Per-instance, not per-fan-out: a process can hold more than one
1789
+ # Client (a second workspace, a test harness, a gem that builds its
1790
+ # own). One of them failing to rebuild — thread exhaustion, a
1791
+ # customer logger that raises — must not cost every client behind it
1792
+ # in the registry its rebuild and leave the child silently dark.
1793
+ Quonfig::Client.each_instance do |client|
1794
+ client.after_fork_in_child
1795
+ rescue StandardError => e
1796
+ Quonfig::Client::LOG.error 'Quonfig fork rebuild failed for one client ' \
1797
+ "(continuing with the rest): #{e.class}: #{e.message}"
1798
+ end
1799
+ end
1306
1800
  pid
1307
1801
  rescue StandardError => e
1308
1802
  # Fork-hook failures must never break the customer's fork. Worst case
1309
- # the child inherits dead SSE threads (the pre-qfg-ryov behavior)
1310
- # bad, but recoverable. Crashing the fork itself is not.
1803
+ # the child holds dropped references and no live threads bad, but
1804
+ # recoverable. Crashing the fork itself is not.
1311
1805
  Quonfig::Client::LOG.error "Quonfig fork hook error: #{e.class}: #{e.message}"
1312
1806
  raise if pid.nil? # super never returned — propagate fork failures
1313
1807