bitfab 0.57.7 → 0.57.8

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b92a57be2a0c05885f0f84f757ede302ab14aed53ad422c85ee2781591a39786
-  data.tar.gz: ddb9b050e2f66e9591c17be161c2ebd25d395b625704baee80e396801f5a9e25
+  metadata.gz: 187464e848086662f6d74efe91dfe95476bdae7284d7ba5668f49b2a4af4c606
+  data.tar.gz: 16fce40beaf7dc062e5d6d31624d6393730847b1eaee578b60459b5f0d27897a
 SHA512:
-  metadata.gz: 0542a3b4dcb64f183ce35f7c9f16ead26e858fa1fefb7889115842e552b4855842ba175bea19893912ec37547708403d115259a201b2e02a1e6cb5ffa84f79f6
-  data.tar.gz: a530cd08bc1cdbc42467ea640c8f64e3035569c6723032626284bd6397744a21f5149ef0bc2d32af50aab1b1bbb856e8ba7510d1eb3640824de4f6f4e4472c43
+  metadata.gz: 7a5f7f9cac1900dca07228819c1c40965cf020deb5adba9f9827c36c76265911e79c5ce3aca16db5e486c354a436f63e9ac487896af9c2cf522581886424887d
+  data.tar.gz: 560d27de6a0766f9625b82bcf33053ebfffd3792dbaf2a21cd59ddd7a129742d265965d086d68138566dee51a5ba02893fdcf0bf67a95ebc8d24bc4d37ecf6ff

data/README.md

@@ -235,7 +235,7 @@ class OrderPipeline
 end
 ```
 
-Capture uses `TracePoint` only during the root call. It skips Ruby/runtime code, dependency gems, Bitfab SDK internals, blocks, lambdas, and forwarding wrappers. When the root belongs to an installed application gem, that gem is the first-party boundary; neighboring dependency gems remain excluded. It follows recursion, modules, inheritance, `define_method`, and same-thread child Fibers; aliases are recorded and excluded by the name that was invoked. Child Fibers created during capture inherit node policies, trace identity, parenting, and depth limits while retaining separate call stacks. Pre-existing Fibers, `Fiber.new(storage: nil)`, and child threads do not inherit subtree context. When an inner capture ends, child Fibers retain any still-active outer capture; inherited context expires once all enclosing captures end. While a root remains active, hitting a span limit stops new capture but allows already-open spans in every Fiber to finish. `max_depth:` (default `30`), `max_captured_subtree_spans:` (default `512`), `max_spans:` (default `2048`), `exclude:`, and `include_wrappers:` control the subtree. `max_spans:` is a strict cap on the total number of spans one trace root sends. It counts the root span itself, discovered calls, `bitfab_node` calls, nested trace roots, and spans whose capture is on or off, including spans sent without inputs and outputs because the sim plan could not be read. Nothing past it is sent. It must be at least 1, and a lower value raises `ArgumentError`. `max_captured_subtree_spans:` caps discovered calls sent with their inputs and outputs. It does not count `bitfab_node` calls, nested trace roots, or spans the loaded sim plan turned capture off for, so those keep recording past it up to `max_spans:`. Spans sent without inputs and outputs only because the sim plan could not be read still count toward it. Past it, further discovered calls with capture on record nothing. It is never higher than `max_spans:`. Once the plan has loaded, the SDK never builds the inputs or outputs of a span whose capture it turned off. The client starts reading the sim plan when it is built. A `bitfab_trace` root that starts while that first read is still running waits for it, for at most five seconds. Once the first read has finished, whether it succeeded or failed, no later root waits. Nested roots and the calls beneath a root never wait, and nothing waits when the plan is disabled. If that first read fails, spans other than each trace's root span are sent without inputs and outputs until a read succeeds, and they still count toward both limits. Automatically captured calls are not replay mock targets. Declare a call with `bitfab_node` when replay must substitute it. The configured node records once in each enclosing trace. Mixing explicit spans with subtree tracing raises `Bitfab::MixedTracingError`.
+Capture uses `TracePoint` only during the root call. It skips Ruby/runtime code, dependency gems, Bitfab SDK internals, blocks, lambdas, and forwarding wrappers. When the root belongs to an installed application gem, that gem is the first-party boundary; neighboring dependency gems remain excluded. It follows recursion, modules, inheritance, `define_method`, and same-thread child Fibers; aliases are recorded and excluded by the name that was invoked. Child Fibers created during capture inherit node policies, trace identity, parenting, and depth limits while retaining separate call stacks. Pre-existing Fibers, `Fiber.new(storage: nil)`, and child threads do not inherit subtree context. When an inner capture ends, child Fibers retain any still-active outer capture; inherited context expires once all enclosing captures end. While a root remains active, hitting a span limit stops new capture but allows already-open spans in every Fiber to finish. `max_depth:` (default `30`), `max_captured_subtree_spans:` (default `500`), `max_spans:` (default `10000`), `exclude:`, and `include_wrappers:` control the subtree. `max_spans:` is a strict cap on the total number of spans one trace root sends. It counts the root span itself, discovered calls, `bitfab_node` calls, nested trace roots, and spans whose capture is on or off, including spans sent without inputs and outputs because the sim plan could not be read. Nothing past it is sent. It must be at least 1, and a lower value raises `ArgumentError`. `max_captured_subtree_spans:` caps discovered calls sent with their inputs and outputs. It does not count `bitfab_node` calls, nested trace roots, or spans the loaded sim plan turned capture off for, so those keep recording past it up to `max_spans:`. Spans sent without inputs and outputs only because the sim plan could not be read still count toward it. Past it, further discovered calls with capture on record nothing. It is never higher than `max_spans:`. Once the plan has loaded, the SDK never builds the inputs or outputs of a span whose capture it turned off. The client starts reading the sim plan when it is built. A `bitfab_trace` root that starts while that first read is still running waits for it, for at most five seconds. Once the first read has finished, whether it succeeded or failed, no later root waits. Nested roots and the calls beneath a root never wait, and nothing waits when the plan is disabled. If that first read fails, spans other than each trace's root span are sent without inputs and outputs until a read succeeds, and they still count toward both limits. Automatically captured calls are not replay mock targets. Declare a call with `bitfab_node` when replay must substitute it. The configured node records once in each enclosing trace. Mixing explicit spans with subtree tracing raises `Bitfab::MixedTracingError`.
 
 When capture ends, already-open automatic calls in suspended Fibers are emitted once with a nil output and `Subtree capture ended before the call returned` as the error, unless a real error was already observed. Their inputs and parenting are retained. The SDK does not resume Fibers, and later resumption cannot add spans to the ended capture. If hot reload moves the root to an unresolved source, the last valid first-party boundary and its exclusions remain in use.
 

data/lib/bitfab/otel.rb

@@ -24,7 +24,7 @@ module Bitfab
     EXPORT_CONCURRENCY_ENV = "BITFAB_OTEL_EXPORT_CONCURRENCY"
     MAX_EXPORT_REQUEST_BYTES = 3_000_000
     MAX_DECOMPRESSED_REQUEST_BYTES = 8_000_000
-    MAX_QUEUE_SIZE = 8_192
+    MAX_QUEUE_SIZE = 16_384
     DIRECT_MAX_EXPORT_BATCH_SIZE = 512
     DIRECT_MAX_REQUEST_BATCH_SIZE = 128
     DEFAULT_EXPORT_CONCURRENCY = 32
@@ -331,17 +331,17 @@ module Bitfab
         if e.oversized
           warn_oversized(batch.spans.size)
         else
-          Bitfab.warn_always("failed to export an OpenTelemetry span batch: #{e.message}")
+          Bitfab.warn_delivery_problem("failed to export an OpenTelemetry span batch: #{e.message}")
         end
         false
       rescue => e
-        Bitfab.warn_always("failed to export an OpenTelemetry span batch: #{e.message}")
+        Bitfab.warn_delivery_problem("failed to export an OpenTelemetry span batch: #{e.message}")
         false
       end
 
       def warn_oversized(span_count)
         subject = (span_count == 1) ? "a single OpenTelemetry span" : "an OpenTelemetry span batch"
-        Bitfab.warn_always("#{subject} exceeded the ingestion request limit and could not be exported")
+        Bitfab.warn_delivery_problem("#{subject} exceeded the ingestion request limit and could not be exported")
       end
 
       def send_with_retries(request)

data/lib/bitfab/replay.rb

@@ -1179,7 +1179,8 @@ module Bitfab
     # repeat itself. Acks cannot settle a request that timed out client-side
     # after the server committed, so anything short of fully delivered falls
     # through to the server poll, which stays the authority there.
-    def wait_for_replay_persistence(http_client, test_run_id, sdk_trace_ids)
+    def wait_for_replay_persistence(http_client, test_run_id, sdk_trace_ids, timeout: nil)
+      timeout ||= PERSISTENCE_TIMEOUT_SECONDS
       trace_ids = sdk_trace_ids.compact
 
       # Checked before flushing: a run whose traces never closed submitted
@@ -1194,7 +1195,7 @@ module Bitfab
       # timeout can fire while requests are still in flight and the server goes
       # on to persist every one of them. Failing here failed runs whose data had
       # fully landed.
-      flushed = Bitfab.flush_traces(timeout: PERSISTENCE_TIMEOUT_SECONDS)
+      flushed = Bitfab.flush_traces(timeout:)
 
       raw_deliveries = http_client.take_trace_deliveries(trace_ids)
       # Server-assigned traces.id per replay trace, read back off the ingest
@@ -1206,7 +1207,7 @@ module Bitfab
       expected_span_counts = deliveries.transform_values(&:span_count)
       return read_back_trace_ids if expected_span_counts.empty? || deliveries.each_value.all?(&:delivered)
 
-      deadline = Otel.monotonic_now + PERSISTENCE_TIMEOUT_SECONDS
+      deadline = Otel.monotonic_now + timeout
       missing = expected_span_counts.keys
 
       while missing.any?

data/lib/bitfab/replay_concurrency.rb

@@ -3,9 +3,9 @@
 module Bitfab
   class ReplayConcurrency
     UNSET = Object.new.freeze
-    attr_reader :attempts, :max_concurrency, :primitive, :memory_throttle, :on_item_finish_in_child_process, :child_timeout
+    attr_reader :attempts, :max_concurrency, :primitive, :memory_throttle, :on_item_finish_in_child_process, :child_timeout, :child_delivery_timeout
 
-    def initialize(attempts: 1, max_concurrency: UNSET, primitive: "async", memory_throttle: UNSET, on_item_finish_in_child_process: nil, child_timeout: 2400)
+    def initialize(attempts: 1, max_concurrency: UNSET, primitive: "async", memory_throttle: UNSET, on_item_finish_in_child_process: nil, child_timeout: 2400, child_delivery_timeout: UNSET)
       raise ArgumentError, "primitive must be async or process" unless %w[async process].include?(primitive)
       raise ArgumentError, "attempts must be an integer from 1 to 100" unless attempts.is_a?(Integer) && attempts.between?(1, 100)
       max_concurrency = (primitive == "process") ? 4 : 10 if max_concurrency.equal?(UNSET)
@@ -24,11 +24,31 @@ module Bitfab
       end
       raise ArgumentError, "child_timeout must be a positive finite number" unless child_timeout.is_a?(Numeric) && child_timeout.finite? && child_timeout.positive?
       @child_timeout = child_timeout
+      @child_delivery_timeout = resolve_child_delivery_timeout(child_delivery_timeout, primitive)
       @attempts, @max_concurrency, @primitive = attempts, max_concurrency, primitive
       @memory_throttle, @on_item_finish_in_child_process = memory_throttle, on_item_finish_in_child_process
       freeze
     end
 
+    private
+
+    def resolve_child_delivery_timeout(timeout, primitive)
+      if timeout.equal?(UNSET)
+        return (primitive == "process") ? Replay::PERSISTENCE_TIMEOUT_SECONDS : nil
+      end
+      return nil if timeout.nil? && primitive != "process"
+      unless timeout.is_a?(Numeric) && timeout.finite? && timeout.positive?
+        raise ArgumentError, "child_delivery_timeout must be a positive number of seconds (got #{timeout.inspect})"
+      end
+      if primitive != "process"
+        raise ArgumentError, "child_delivery_timeout bounds how long a child process waits for its spans to reach Bitfab, " \
+          "and only primitive: process creates one. Under primitive: async the run's own delivery wait applies."
+      end
+      timeout
+    end
+
+    public
+
     def self.resolve(concurrency, attempts, max_concurrency)
       if concurrency
         raise ArgumentError, "concurrency must be a ReplayConcurrency" unless concurrency.is_a?(self)

data/lib/bitfab/replay_processes.rb

@@ -42,37 +42,57 @@ module Bitfab
       end
       http = client.instance_variable_get(:@http_client)
       overrides = MockOverride.normalize(options[:mock_override]) + (client.instance_variable_get(:@mock_overrides) || [])
-      result = Replay.process_single_item(http, assignment.fetch("item"), receiver, method_name,
-        assignment.fetch("test_run_id"), assignment.fetch("mock"), options[:adapt_inputs],
-        assignment.fetch("include_db_branch_lease"), assignment["db_branch_settings"],
-        mock_overrides: overrides, dry_run: assignment.fetch("dry_run"))
-      unless assignment["dry_run"]
-        begin
-          mapped = Replay.wait_for_replay_persistence(http, assignment.fetch("test_run_id"), [result[:_sdk_trace_id]])
-          result[:trace_id] = mapped[result[:_sdk_trace_id]] || result[:trace_id]
-          if result[:_sdk_trace_id] && result[:trace_id].nil?
-            status = http.get_replay_status(assignment.fetch("test_run_id"), {result[:_sdk_trace_id] => 1})
-            result[:trace_id] = status.fetch("traceIds", {})[result[:_sdk_trace_id]]
+      delivery_timeout = options[:concurrency]&.child_delivery_timeout || Replay::PERSISTENCE_TIMEOUT_SECONDS
+      result = nil
+      Bitfab.collect_delivery_problems do
+        result = Replay.process_single_item(http, assignment.fetch("item"), receiver, method_name,
+          assignment.fetch("test_run_id"), assignment.fetch("mock"), options[:adapt_inputs],
+          assignment.fetch("include_db_branch_lease"), assignment["db_branch_settings"],
+          mock_overrides: overrides, dry_run: assignment.fetch("dry_run"))
+        unless assignment["dry_run"]
+          begin
+            mapped = Replay.wait_for_replay_persistence(http, assignment.fetch("test_run_id"), [result[:_sdk_trace_id]], timeout: delivery_timeout)
+            result[:trace_id] = mapped[result[:_sdk_trace_id]] || result[:trace_id]
+            if result[:_sdk_trace_id] && result[:trace_id].nil?
+              status = http.get_replay_status(assignment.fetch("test_run_id"), {result[:_sdk_trace_id] => 1})
+              result[:trace_id] = status.fetch("traceIds", {})[result[:_sdk_trace_id]]
+            end
+            raise "Child replay trace persistence could not be confirmed" if result[:error].nil? && result[:trace_id].nil?
+          rescue => error
+            result[:replay_error] = error
+            result[:error] = error.message
           end
-          raise "Child replay trace persistence could not be confirmed" if result[:error].nil? && result[:trace_id].nil?
-        rescue => error
-          result[:replay_error] = error
-          result[:error] = error.message
         end
-      end
-      write_result(assignment.fetch("result_path"), result)
-      hook = options[:concurrency]&.on_item_finish_in_child_process
-      if hook && !assignment["dry_run"]
-        begin
-          hook.call({test_run_id: assignment.fetch("test_run_id"), item: Replay.public_replay_items([result]).first})
-        rescue => error
-          stderr.puts "[replay] child grading hook failed for #{result[:original_trace_id]} attempt #{result[:attempt]}: #{error}"
+        write_result(assignment.fetch("result_path"), result)
+        hook = options[:concurrency]&.on_item_finish_in_child_process
+        if hook && !assignment["dry_run"]
+          begin
+            hook.call({test_run_id: assignment.fetch("test_run_id"), item: Replay.public_replay_items([result]).first})
+          rescue => error
+            stderr.puts "[replay] child grading hook failed for #{result[:original_trace_id]} attempt #{result[:attempt]}: #{error}"
+          end
         end
+        unless client.close(timeout: delivery_timeout) || assignment["dry_run"]
+          Bitfab.count_delivery_problem("delivery was not confirmed within child_delivery_timeout (#{delivery_timeout}s). " \
+            "Spans that failed or were still sending when the child exited may never reach Bitfab.")
+        end
+        write_delivery_problems(assignment["delivery_problems_path"], result, Bitfab.delivery_problem_counts, stderr)
       end
-      client.close(timeout: 5)
       result
     end
 
+    def write_delivery_problems(path, result, problems, stderr)
+      return if problems.empty?
+      label = "#{result[:original_trace_id]}##{result[:attempt]}"
+      report = problems.map do |message, count|
+        "[replay] span delivery problem for #{label}: #{message}#{" (x#{count})" if count > 1}\n"
+      end.join
+      stderr.write(report)
+      File.write(path, report, mode: "w", perm: 0o600) if path
+    rescue SystemCallError
+      nil
+    end
+
     def write_result(path, result)
       encoded = begin
         Marshal.dump(result)
@@ -116,7 +136,8 @@ module Bitfab
         result_path = File.join(@directory, "#{index}.result")
         assignment_path = File.join(@directory, "#{index}.json")
         stdout_path, stderr_path = File.join(@directory, "#{index}.out"), File.join(@directory, "#{index}.err")
-        assignment = {pipeline: @args.fetch(:pipeline), params: @params, item:, test_run_id:, mock:, include_db_branch_lease:, db_branch_settings:, dry_run:, result_path:}
+        delivery_problems_path = File.join(@directory, "#{index}.delivery")
+        assignment = {pipeline: @args.fetch(:pipeline), params: @params, item:, test_run_id:, mock:, include_db_branch_lease:, db_branch_settings:, dry_run:, result_path:, delivery_problems_path:}
         File.write(assignment_path, JSON.generate(assignment), mode: "w", perm: 0o600)
         executable = File.expand_path("../../exe/bitfab-replay", __dir__)
         pid = @mutex.synchronize do
@@ -154,6 +175,8 @@ module Bitfab
         end
         child_errors = read_tail(stderr_path)
         @mutex.synchronize { @stderr.write(child_errors) } unless child_errors.empty?
+        delivery_problems = File.file?(delivery_problems_path) ? File.read(delivery_problems_path) : ""
+        @mutex.synchronize { @stderr.write(delivery_problems) } unless delivery_problems.empty? || child_errors.include?(delivery_problems)
         unless File.file?(result_path)
           output_tail = read_tail(stdout_path)
           @mutex.synchronize { @stderr.write(output_tail) } unless output_tail.empty?

data/lib/bitfab/replay_registry.rb

@@ -220,7 +220,8 @@ module Bitfab
                            end,
           memory_throttle: args.fetch(:memory_throttle, configured ? configured.memory_throttle : ReplayConcurrency::UNSET),
           on_item_finish_in_child_process: configured&.on_item_finish_in_child_process,
-          child_timeout: configured&.child_timeout || 2400
+          child_timeout: configured&.child_timeout || 2400,
+          child_delivery_timeout: configured&.child_delivery_timeout || ReplayConcurrency::UNSET
         )
         ReplayProcesses.current&.configure(registry, options[:concurrency]) if options[:concurrency].primitive == "process"
       end

data/lib/bitfab/subtree.rb

@@ -10,8 +10,8 @@ module Bitfab
 
   module Subtree
     DEFAULT_MAX_DEPTH = 30
-    DEFAULT_MAX_SPANS = 2048
-    DEFAULT_MAX_CAPTURED_SUBTREE_SPANS = 512
+    DEFAULT_MAX_SPANS = 10_000
+    DEFAULT_MAX_CAPTURED_SUBTREE_SPANS = 500
     LIMIT_ORDER = %i[max_depth max_spans max_captured_subtree_spans].freeze
     SESSION_KEY = :bitfab_subtree_session
     DISPATCHER_KEY = :bitfab_subtree_dispatcher

data/lib/bitfab/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Bitfab
-  VERSION = "0.57.7"
+  VERSION = "0.57.8"
 end

data/lib/bitfab/warn_once.rb

@@ -12,6 +12,8 @@ module Bitfab
   # warns once, not just the first one seen.
   @warn_once_seen = Set.new
   @warn_once_mutex = Mutex.new
+  @delivery_problems = nil
+  @delivery_problems_mutex = Mutex.new
 
   class << self
     def warn_once(key, message)
@@ -37,6 +39,27 @@ module Bitfab
       # Logging must never crash the host app (e.g. a closed $stderr).
     end
 
+    def warn_delivery_problem(message)
+      count_delivery_problem(message)
+      warn_always(message)
+    end
+
+    def count_delivery_problem(message)
+      @delivery_problems_mutex.synchronize { @delivery_problems[message] += 1 if @delivery_problems }
+    end
+
+    def delivery_problem_counts
+      @delivery_problems_mutex.synchronize { @delivery_problems ? @delivery_problems.dup : {} }
+    end
+
+    def collect_delivery_problems
+      problems = Hash.new(0)
+      @delivery_problems_mutex.synchronize { @delivery_problems = problems }
+      yield problems
+    ensure
+      @delivery_problems_mutex.synchronize { @delivery_problems = nil }
+    end
+
     # Test-only: clear the dedup set so a warning can fire again.
     def _reset_warn_once
       @warn_once_mutex.synchronize { @warn_once_seen.clear }

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: bitfab
 version: !ruby/object:Gem::Version
-  version: 0.57.7
+  version: 0.57.8
 platform: ruby
 authors:
 - Harvest Team