bitfab 0.57.6 → 0.57.8

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 34dfb7ca50f4c2dceaa4aa4ed46ee6e9426a5d8051a3602084fee0640a77c9e9
-  data.tar.gz: ad1e112ccdfc9a2cee2860b602e3edf3629686ef6228e81ac4b80f5132ae1fc1
+  metadata.gz: 187464e848086662f6d74efe91dfe95476bdae7284d7ba5668f49b2a4af4c606
+  data.tar.gz: 16fce40beaf7dc062e5d6d31624d6393730847b1eaee578b60459b5f0d27897a
 SHA512:
-  metadata.gz: 764ad6880ee3656d90d28ecb5571298cc54ab2bd1bc6a6b2b07399fd2bee8fd151ae3adcfdbf9f2bdc8f7c0d0fccbb1da29e33c93dc68353e11058be22dc9cc8
-  data.tar.gz: afadb371e2d697bab8aaac2a4f02a57491ab37745b6b756ca3302931915f6fec7eabcc80d4789e79f06a5f5b9dae9ab22627a1fc4f35a7f2f610d175da2c12ef
+  metadata.gz: 7a5f7f9cac1900dca07228819c1c40965cf020deb5adba9f9827c36c76265911e79c5ce3aca16db5e486c354a436f63e9ac487896af9c2cf522581886424887d
+  data.tar.gz: 560d27de6a0766f9625b82bcf33053ebfffd3792dbaf2a21cd59ddd7a129742d265965d086d68138566dee51a5ba02893fdcf0bf67a95ebc8d24bc4d37ecf6ff

data/README.md

@@ -235,7 +235,7 @@ class OrderPipeline
 end
 ```
 
-Capture uses `TracePoint` only during the root call. It skips Ruby/runtime code, dependency gems, Bitfab SDK internals, blocks, lambdas, and forwarding wrappers. When the root belongs to an installed application gem, that gem is the first-party boundary; neighboring dependency gems remain excluded. It follows recursion, modules, inheritance, `define_method`, and same-thread child Fibers; aliases are recorded and excluded by the name that was invoked. Child Fibers created during capture inherit node policies, trace identity, parenting, and depth limits while retaining separate call stacks. Pre-existing Fibers, `Fiber.new(storage: nil)`, and child threads do not inherit subtree context. When an inner capture ends, child Fibers retain any still-active outer capture; inherited context expires once all enclosing captures end. While a root remains active, hitting a span limit stops new capture but allows already-open spans in every Fiber to finish. `max_depth:` (default `30`), `max_captured_subtree_spans:` (default `512`), `max_spans:` (default `2048`), `exclude:`, and `include_wrappers:` control the subtree. `max_spans:` is a strict cap on the total number of spans one trace root sends. It counts the root span itself, discovered calls, `bitfab_node` calls, nested trace roots, and spans whose capture is on or off, including spans sent without inputs and outputs because the sim plan could not be read. Nothing past it is sent. It must be at least 1, and a lower value raises `ArgumentError`. `max_captured_subtree_spans:` caps discovered calls sent with their inputs and outputs. It does not count `bitfab_node` calls, nested trace roots, or spans the loaded sim plan turned capture off for, so those keep recording past it up to `max_spans:`. Spans sent without inputs and outputs only because the sim plan could not be read still count toward it. Past it, further discovered calls with capture on record nothing. It is never higher than `max_spans:`. Once the plan has loaded, the SDK never builds the inputs or outputs of a span whose capture it turned off. The client starts reading the sim plan when it is built. A `bitfab_trace` root that starts while that first read is still running waits for it, for at most five seconds. Once the first read has finished, whether it succeeded or failed, no later root waits. Nested roots and the calls beneath a root never wait, and nothing waits when the plan is disabled. If that first read fails, spans other than each trace's root span are sent without inputs and outputs until a read succeeds, and they still count toward both limits. Automatically captured calls are not replay mock targets. Declare a call with `bitfab_node` when replay must substitute it. The configured node records once in each enclosing trace. Mixing explicit spans with subtree tracing raises `Bitfab::MixedTracingError`.
+Capture uses `TracePoint` only during the root call. It skips Ruby/runtime code, dependency gems, Bitfab SDK internals, blocks, lambdas, and forwarding wrappers. When the root belongs to an installed application gem, that gem is the first-party boundary; neighboring dependency gems remain excluded. It follows recursion, modules, inheritance, `define_method`, and same-thread child Fibers; aliases are recorded and excluded by the name that was invoked. Child Fibers created during capture inherit node policies, trace identity, parenting, and depth limits while retaining separate call stacks. Pre-existing Fibers, `Fiber.new(storage: nil)`, and child threads do not inherit subtree context. When an inner capture ends, child Fibers retain any still-active outer capture; inherited context expires once all enclosing captures end. While a root remains active, hitting a span limit stops new capture but allows already-open spans in every Fiber to finish. `max_depth:` (default `30`), `max_captured_subtree_spans:` (default `500`), `max_spans:` (default `10000`), `exclude:`, and `include_wrappers:` control the subtree. `max_spans:` is a strict cap on the total number of spans one trace root sends. It counts the root span itself, discovered calls, `bitfab_node` calls, nested trace roots, and spans whose capture is on or off, including spans sent without inputs and outputs because the sim plan could not be read. Nothing past it is sent. It must be at least 1, and a lower value raises `ArgumentError`. `max_captured_subtree_spans:` caps discovered calls sent with their inputs and outputs. It does not count `bitfab_node` calls, nested trace roots, or spans the loaded sim plan turned capture off for, so those keep recording past it up to `max_spans:`. Spans sent without inputs and outputs only because the sim plan could not be read still count toward it. Past it, further discovered calls with capture on record nothing. It is never higher than `max_spans:`. Once the plan has loaded, the SDK never builds the inputs or outputs of a span whose capture it turned off. The client starts reading the sim plan when it is built. A `bitfab_trace` root that starts while that first read is still running waits for it, for at most five seconds. Once the first read has finished, whether it succeeded or failed, no later root waits. Nested roots and the calls beneath a root never wait, and nothing waits when the plan is disabled. If that first read fails, spans other than each trace's root span are sent without inputs and outputs until a read succeeds, and they still count toward both limits. Automatically captured calls are not replay mock targets. Declare a call with `bitfab_node` when replay must substitute it. The configured node records once in each enclosing trace. Mixing explicit spans with subtree tracing raises `Bitfab::MixedTracingError`.
 
 When capture ends, already-open automatic calls in suspended Fibers are emitted once with a nil output and `Subtree capture ended before the call returned` as the error, unless a real error was already observed. Their inputs and parenting are retained. The SDK does not resume Fibers, and later resumption cannot add spans to the ended capture. If hot reload moves the root to an unresolved source, the last valid first-party boundary and its exclusions remain in use.
 

data/lib/bitfab/baml.rb

@@ -170,7 +170,6 @@ module Bitfab
   end
 
   class Client
-    # Execute a stored BAML prompt locally and return its parsed output.
     def call(method_name, **inputs)
       version = @http_client.request("/api/sdk/functions/lookup", {"name" => method_name})
       raise ArgumentError, "Function #{method_name.inspect} not found. Create it at #{@service_url}/functions" unless version["id"]
@@ -182,7 +181,9 @@ module Bitfab
       rescue
         result.to_s
       end
-      @http_client.send_internal_trace(version["id"], {"result" => output, "inputs" => inputs, "rawCollector" => collector, "source" => "ruby-sdk"})
+      if should_record? && !active_span_content_withheld?
+        @http_client.send_internal_trace(version["id"], {"result" => output, "inputs" => inputs, "rawCollector" => collector, "source" => "ruby-sdk"})
+      end
       result
     end
 

data/lib/bitfab/client.rb

@@ -49,14 +49,10 @@ module Bitfab
       @enabled_in_code = capture_enabled.nil? ? enabled : capture_enabled
       @capture_configured_from_env = nil
       @strict = strict
-      # Cached only once a non-empty key is found, so an early resolve (before
-      # env loaded) can't poison a later one.
       @resolved_api_key = nil
       @api_key_warned = false
-      # The key is NOT read here. HttpClient gets a proc so the key is resolved
-      # at send time, after any env loading has run.
       @http_client = HttpClient.new(api_key: -> { resolve_api_key }, service_url: @service_url, timeout:)
-      @http_client.configure_simulation_plan(enabled: simulation_plan)
+      @http_client.configure_simulation_plan(enabled: simulation_plan, start: !api_key_resolved_lazily?)
       @datasets = Datasets.new(@http_client)
       @assertion_categories = AssertionCategories.new(@http_client)
       @traces = Traces.new(@http_client)
@@ -381,11 +377,10 @@ module Bitfab
       explicit_span_receiver: nil, explicit_span_method_name: nil, test_run_id: nil, finalize: nil,
       surface: nil, on_start: nil, on_complete: nil, around_body: nil, span_data: {}, managed_root: false,
       result_recorded_elsewhere: false)
-      # Decide at CALL time, not construction. The key may be set after the
-      # client is built (env loaded later), so re-checking per call lets a
-      # late-resolved key take effect.
       return yield unless should_record?
 
+      @http_client.refresh_simulation_plan
+
       resolved_capture_when = capture_when.to_s
       unless ["always", "nested"].include?(resolved_capture_when)
         Bitfab.warn_once(
@@ -513,6 +508,12 @@ module Bitfab
       completed = false
       @http_client.trace_completion.start(trace_id, span_id)
       recording_override = managed_root ? {} : nil
+      content_withheld = lambda do
+        instrumentation = recording_override&.dig(:instrumentation) || ((surface == :trace) ? "trace" : "span")
+        content_off_by_simulation_plan?(
+          trace_id:, trace_function_key:, span_name: recording_override&.dig(:name) || span_name, instrumentation:
+        ) || (!parent_span_id.nil? && simulation_plan_unavailable?(instrumentation:))
+      end
 
       complete = lambda do |final_result, final_error|
         # Never crash the host app due to span building/sending. Idempotent:
@@ -528,9 +529,7 @@ module Bitfab
           recorded_error = final_error
           recorded_name = recording_override&.dig(:name) || span_name
           recorded_instrumentation = recording_override&.dig(:instrumentation) || ((surface == :trace) ? "trace" : "span")
-          content_off = content_off_by_simulation_plan?(
-            trace_id:, trace_function_key:, span_name: recorded_name, instrumentation: recorded_instrumentation
-          ) || (!parent_span_id.nil? && simulation_plan_unavailable?(instrumentation: recorded_instrumentation))
+          content_off = content_withheld.call
           if finalize && final_error.nil? && !(content_off && !result_recorded_elsewhere)
             begin
               recorded_result = finalize.call(final_result)
@@ -613,7 +612,7 @@ module Bitfab
             span_id:,
             explicit_span_receiver:,
             explicit_span_method_name: explicit_span_method_name&.to_sym,
-            surface:, recording_override:
+            surface:, recording_override:, content_withheld:
           ) do
             invoke = -> { around_body ? around_body.call { yield } : yield }
             result = if explicit_span_receiver
@@ -704,6 +703,18 @@ module Bitfab
       nil
     end
 
+    def active_span_content_withheld?
+      return true if Subtree.current_session&.current_span_content_withheld?
+
+      SpanContext.current&.dig(:content_withheld)&.call == true
+    rescue
+      true
+    end
+
+    def api_key_resolved_lazily?
+      @api_key_config.respond_to?(:call) || @api_key_config.to_s.strip.empty?
+    end
+
     # Resolve the API key lazily, the first time a span actually needs it.
     #
     # The key is intentionally NOT read at construction: a client built when a
@@ -955,7 +966,7 @@ module Bitfab
         span_data["error_source"] = "code"
       end
       span_data["contexts"] = contexts if contexts&.any?
-      span_data["prompt"] = prompt if prompt
+      span_data["prompt"] = prompt if prompt && !content_off
 
       raw_span = {
         "id" => span_id,

data/lib/bitfab/connection_pool.rb

@@ -0,0 +1,136 @@
+# frozen_string_literal: true
+
+require "net/http"
+
+module Bitfab
+  class ConnectionPool
+    KEEP_ALIVE_TIMEOUT_SECONDS = 30
+    MAX_IDLE_CONNECTIONS = 64
+
+    IdleConnection = Struct.new(:http, :released_at)
+    private_constant :IdleConnection
+
+    @registry = ObjectSpace::WeakMap.new
+    @registry_mutex = Mutex.new
+
+    class << self
+      def register(pool)
+        @registry_mutex.synchronize { @registry[pool] = true }
+      end
+
+      def close_all
+        pools = @registry_mutex.synchronize { @registry.keys }
+        pools.each(&:close)
+      end
+    end
+
+    def initialize
+      @pid = Process.pid
+      @mutex = Mutex.new
+      @idle = {}
+      self.class.register(self)
+    end
+
+    def with_connection(uri, timeout)
+      key = [uri.scheme, uri.host, uri.port]
+      owner_pid = Process.pid
+      reusable = false
+      http = checkout(key, uri, timeout)
+      response = yield http
+      reusable = true
+      response
+    ensure
+      release(key, http, owner_pid, reusable) if http
+    end
+
+    def close
+      drain.each { |http| finish(http) }
+    end
+
+    def idle_count
+      reset_after_fork
+      @mutex.synchronize { @idle.sum { |_key, connections| connections.size } }
+    end
+
+    private
+
+    def checkout(key, uri, timeout)
+      http = take_idle(key) || connect(uri, timeout)
+      http.read_timeout = timeout
+      http.write_timeout = timeout
+      http
+    end
+
+    def connect(uri, timeout)
+      http = Net::HTTP.new(uri.host, uri.port)
+      http.use_ssl = uri.scheme == "https"
+      http.open_timeout = timeout
+      http.read_timeout = timeout
+      http.write_timeout = timeout
+      http.keep_alive_timeout = KEEP_ALIVE_TIMEOUT_SECONDS
+      http.start
+    end
+
+    def take_idle(key)
+      reset_after_fork
+      now = monotonic_now
+      expired = []
+      http = @mutex.synchronize do
+        connections = @idle[key] || []
+        stale, fresh = connections.partition { |idle| now - idle.released_at >= KEEP_ALIVE_TIMEOUT_SECONDS }
+        expired.concat(stale.map(&:http))
+        @idle[key] = fresh
+        fresh.pop&.http
+      end
+      expired.each { |stale_http| finish(stale_http) }
+      http
+    end
+
+    def release(key, http, owner_pid, reusable)
+      return unless owner_pid == Process.pid
+
+      unless reusable && http.started?
+        finish(http)
+        return
+      end
+
+      reset_after_fork
+      pooled = @mutex.synchronize do
+        next false if @idle.sum { |_key, connections| connections.size } >= MAX_IDLE_CONNECTIONS
+
+        (@idle[key] ||= []) << IdleConnection.new(http, monotonic_now)
+        true
+      end
+      finish(http) unless pooled
+    end
+
+    def drain
+      return [] if reset_after_fork
+
+      @mutex.synchronize do
+        connections = @idle.values.flatten.map(&:http)
+        @idle = {}
+        connections
+      end
+    end
+
+    def reset_after_fork
+      return false if @pid == Process.pid
+
+      @pid = Process.pid
+      @mutex = Mutex.new
+      @idle = {}
+      true
+    end
+
+    def finish(http)
+      http.finish if http.started?
+    rescue
+      nil
+    end
+
+    def monotonic_now
+      Process.clock_gettime(Process::CLOCK_MONOTONIC)
+    end
+  end
+end

data/lib/bitfab/http_client.rb

@@ -6,6 +6,7 @@ require "uri"
 
 require_relative "git_state"
 require_relative "compress"
+require_relative "connection_pool"
 require_relative "constants"
 require_relative "serialize"
 require_relative "transport"
@@ -37,6 +38,7 @@ module Bitfab
       @api_key = api_key
       @service_url = (service_url || DEFAULT_SERVICE_URL).chomp("/")
       @timeout = timeout
+      @connections = ConnectionPool.new
       @transport_mutex = Mutex.new
       # Eager for the same reason the transport's mutex is: `||=` racing between
       # the export threads recording acks would hand each a different mutex.
@@ -53,9 +55,9 @@ module Bitfab
       @close_condition = ConditionVariable.new
     end
 
-    def configure_simulation_plan(enabled: true)
+    def configure_simulation_plan(enabled: true, start: true)
       @simulation_plan = SimulationPlan.new(self, enabled:)
-      @simulation_plan.refresh
+      @simulation_plan.refresh if start
       @simulation_plan
     end
 
@@ -80,7 +82,7 @@ module Bitfab
     end
 
     def get_simulation_plan
-      get("/api/sdk/sim-plan", timeout: SimulationPlan::READ_TIMEOUT, connection_close: true)
+      get("/api/sdk/sim-plan", timeout: SimulationPlan::READ_TIMEOUT)
     end
 
     # Flush and permanently close this client's tracing transport.
@@ -112,6 +114,7 @@ module Bitfab
         shutdown = transport.nil? || transport.shutdown(remaining)
         result = released && shutdown
       ensure
+        @connections.close
         @transport_mutex.synchronize do
           @close_result = result
           @close_condition.broadcast
@@ -157,16 +160,11 @@ module Bitfab
       last_error = nil
 
       max_retries.times do |attempt|
-        http = Net::HTTP.new(uri.host, uri.port)
-        http.use_ssl = uri.scheme == "https"
-        http.open_timeout = request_timeout
-        http.read_timeout = request_timeout
-
         req = Net::HTTPGenericRequest.new(method, true, true, uri.request_uri, headers)
         req["Content-Encoding"] = prepared.content_encoding if prepared.content_encoding
         req.body = prepared.body
 
-        response = http.request(req)
+        response = @connections.with_connection(uri, request_timeout) { |http| http.request(req) }
 
         unless response.is_a?(Net::HTTPSuccess)
           raise Net::HTTPError.new("HTTP #{response.code}: #{response.body}", response)
@@ -216,19 +214,12 @@ module Bitfab
 
     # Make a GET request to the Bitfab API.
     # Returns parsed JSON response hash.
-    def get(endpoint, timeout: nil, connection_close: false)
+    def get(endpoint, timeout: nil)
       uri = URI("#{@service_url}#{endpoint}")
       request_timeout = timeout || @timeout
 
-      http = Net::HTTP.new(uri.host, uri.port)
-      http.use_ssl = uri.scheme == "https"
-      http.open_timeout = request_timeout
-      http.read_timeout = request_timeout
-
-      # request_uri (not path) so any query string on the endpoint survives.
       req = Net::HTTP::Get.new(uri.request_uri, headers)
-      req["Connection"] = "close" if connection_close
-      response = http.request(req)
+      response = @connections.with_connection(uri, request_timeout) { |http| http.request(req) }
 
       unless response.is_a?(Net::HTTPSuccess)
         raise Net::HTTPError.new("HTTP #{response.code}: #{response.body}", response)
@@ -742,5 +733,6 @@ module Bitfab
   at_exit do
     SimulationPlan.release_all(2, stop: true)
     Transport.shutdown_trace_transports(2)
+    ConnectionPool.close_all
   end
 end

data/lib/bitfab/otel.rb

@@ -24,7 +24,7 @@ module Bitfab
     EXPORT_CONCURRENCY_ENV = "BITFAB_OTEL_EXPORT_CONCURRENCY"
     MAX_EXPORT_REQUEST_BYTES = 3_000_000
     MAX_DECOMPRESSED_REQUEST_BYTES = 8_000_000
-    MAX_QUEUE_SIZE = 8_192
+    MAX_QUEUE_SIZE = 16_384
     DIRECT_MAX_EXPORT_BATCH_SIZE = 512
     DIRECT_MAX_REQUEST_BATCH_SIZE = 128
     DEFAULT_EXPORT_CONCURRENCY = 32
@@ -331,17 +331,17 @@ module Bitfab
         if e.oversized
           warn_oversized(batch.spans.size)
         else
-          Bitfab.warn_always("failed to export an OpenTelemetry span batch: #{e.message}")
+          Bitfab.warn_delivery_problem("failed to export an OpenTelemetry span batch: #{e.message}")
         end
         false
       rescue => e
-        Bitfab.warn_always("failed to export an OpenTelemetry span batch: #{e.message}")
+        Bitfab.warn_delivery_problem("failed to export an OpenTelemetry span batch: #{e.message}")
         false
       end
 
       def warn_oversized(span_count)
         subject = (span_count == 1) ? "a single OpenTelemetry span" : "an OpenTelemetry span batch"
-        Bitfab.warn_always("#{subject} exceeded the ingestion request limit and could not be exported")
+        Bitfab.warn_delivery_problem("#{subject} exceeded the ingestion request limit and could not be exported")
       end
 
       def send_with_retries(request)

data/lib/bitfab/replay.rb

@@ -1179,7 +1179,8 @@ module Bitfab
     # repeat itself. Acks cannot settle a request that timed out client-side
     # after the server committed, so anything short of fully delivered falls
     # through to the server poll, which stays the authority there.
-    def wait_for_replay_persistence(http_client, test_run_id, sdk_trace_ids)
+    def wait_for_replay_persistence(http_client, test_run_id, sdk_trace_ids, timeout: nil)
+      timeout ||= PERSISTENCE_TIMEOUT_SECONDS
       trace_ids = sdk_trace_ids.compact
 
       # Checked before flushing: a run whose traces never closed submitted
@@ -1194,7 +1195,7 @@ module Bitfab
       # timeout can fire while requests are still in flight and the server goes
       # on to persist every one of them. Failing here failed runs whose data had
       # fully landed.
-      flushed = Bitfab.flush_traces(timeout: PERSISTENCE_TIMEOUT_SECONDS)
+      flushed = Bitfab.flush_traces(timeout:)
 
       raw_deliveries = http_client.take_trace_deliveries(trace_ids)
       # Server-assigned traces.id per replay trace, read back off the ingest
@@ -1206,7 +1207,7 @@ module Bitfab
       expected_span_counts = deliveries.transform_values(&:span_count)
       return read_back_trace_ids if expected_span_counts.empty? || deliveries.each_value.all?(&:delivered)
 
-      deadline = Otel.monotonic_now + PERSISTENCE_TIMEOUT_SECONDS
+      deadline = Otel.monotonic_now + timeout
       missing = expected_span_counts.keys
 
       while missing.any?

data/lib/bitfab/replay_concurrency.rb

@@ -3,9 +3,9 @@
 module Bitfab
   class ReplayConcurrency
     UNSET = Object.new.freeze
-    attr_reader :attempts, :max_concurrency, :primitive, :memory_throttle, :on_item_finish_in_child_process, :child_timeout
+    attr_reader :attempts, :max_concurrency, :primitive, :memory_throttle, :on_item_finish_in_child_process, :child_timeout, :child_delivery_timeout
 
-    def initialize(attempts: 1, max_concurrency: UNSET, primitive: "async", memory_throttle: UNSET, on_item_finish_in_child_process: nil, child_timeout: 2400)
+    def initialize(attempts: 1, max_concurrency: UNSET, primitive: "async", memory_throttle: UNSET, on_item_finish_in_child_process: nil, child_timeout: 2400, child_delivery_timeout: UNSET)
       raise ArgumentError, "primitive must be async or process" unless %w[async process].include?(primitive)
       raise ArgumentError, "attempts must be an integer from 1 to 100" unless attempts.is_a?(Integer) && attempts.between?(1, 100)
       max_concurrency = (primitive == "process") ? 4 : 10 if max_concurrency.equal?(UNSET)
@@ -24,11 +24,31 @@ module Bitfab
       end
       raise ArgumentError, "child_timeout must be a positive finite number" unless child_timeout.is_a?(Numeric) && child_timeout.finite? && child_timeout.positive?
       @child_timeout = child_timeout
+      @child_delivery_timeout = resolve_child_delivery_timeout(child_delivery_timeout, primitive)
       @attempts, @max_concurrency, @primitive = attempts, max_concurrency, primitive
       @memory_throttle, @on_item_finish_in_child_process = memory_throttle, on_item_finish_in_child_process
       freeze
     end
 
+    private
+
+    def resolve_child_delivery_timeout(timeout, primitive)
+      if timeout.equal?(UNSET)
+        return (primitive == "process") ? Replay::PERSISTENCE_TIMEOUT_SECONDS : nil
+      end
+      return nil if timeout.nil? && primitive != "process"
+      unless timeout.is_a?(Numeric) && timeout.finite? && timeout.positive?
+        raise ArgumentError, "child_delivery_timeout must be a positive number of seconds (got #{timeout.inspect})"
+      end
+      if primitive != "process"
+        raise ArgumentError, "child_delivery_timeout bounds how long a child process waits for its spans to reach Bitfab, " \
+          "and only primitive: process creates one. Under primitive: async the run's own delivery wait applies."
+      end
+      timeout
+    end
+
+    public
+
     def self.resolve(concurrency, attempts, max_concurrency)
       if concurrency
         raise ArgumentError, "concurrency must be a ReplayConcurrency" unless concurrency.is_a?(self)

data/lib/bitfab/replay_processes.rb

@@ -42,37 +42,57 @@ module Bitfab
       end
       http = client.instance_variable_get(:@http_client)
       overrides = MockOverride.normalize(options[:mock_override]) + (client.instance_variable_get(:@mock_overrides) || [])
-      result = Replay.process_single_item(http, assignment.fetch("item"), receiver, method_name,
-        assignment.fetch("test_run_id"), assignment.fetch("mock"), options[:adapt_inputs],
-        assignment.fetch("include_db_branch_lease"), assignment["db_branch_settings"],
-        mock_overrides: overrides, dry_run: assignment.fetch("dry_run"))
-      unless assignment["dry_run"]
-        begin
-          mapped = Replay.wait_for_replay_persistence(http, assignment.fetch("test_run_id"), [result[:_sdk_trace_id]])
-          result[:trace_id] = mapped[result[:_sdk_trace_id]] || result[:trace_id]
-          if result[:_sdk_trace_id] && result[:trace_id].nil?
-            status = http.get_replay_status(assignment.fetch("test_run_id"), {result[:_sdk_trace_id] => 1})
-            result[:trace_id] = status.fetch("traceIds", {})[result[:_sdk_trace_id]]
+      delivery_timeout = options[:concurrency]&.child_delivery_timeout || Replay::PERSISTENCE_TIMEOUT_SECONDS
+      result = nil
+      Bitfab.collect_delivery_problems do
+        result = Replay.process_single_item(http, assignment.fetch("item"), receiver, method_name,
+          assignment.fetch("test_run_id"), assignment.fetch("mock"), options[:adapt_inputs],
+          assignment.fetch("include_db_branch_lease"), assignment["db_branch_settings"],
+          mock_overrides: overrides, dry_run: assignment.fetch("dry_run"))
+        unless assignment["dry_run"]
+          begin
+            mapped = Replay.wait_for_replay_persistence(http, assignment.fetch("test_run_id"), [result[:_sdk_trace_id]], timeout: delivery_timeout)
+            result[:trace_id] = mapped[result[:_sdk_trace_id]] || result[:trace_id]
+            if result[:_sdk_trace_id] && result[:trace_id].nil?
+              status = http.get_replay_status(assignment.fetch("test_run_id"), {result[:_sdk_trace_id] => 1})
+              result[:trace_id] = status.fetch("traceIds", {})[result[:_sdk_trace_id]]
+            end
+            raise "Child replay trace persistence could not be confirmed" if result[:error].nil? && result[:trace_id].nil?
+          rescue => error
+            result[:replay_error] = error
+            result[:error] = error.message
           end
-          raise "Child replay trace persistence could not be confirmed" if result[:error].nil? && result[:trace_id].nil?
-        rescue => error
-          result[:replay_error] = error
-          result[:error] = error.message
         end
-      end
-      write_result(assignment.fetch("result_path"), result)
-      hook = options[:concurrency]&.on_item_finish_in_child_process
-      if hook && !assignment["dry_run"]
-        begin
-          hook.call({test_run_id: assignment.fetch("test_run_id"), item: Replay.public_replay_items([result]).first})
-        rescue => error
-          stderr.puts "[replay] child grading hook failed for #{result[:original_trace_id]} attempt #{result[:attempt]}: #{error}"
+        write_result(assignment.fetch("result_path"), result)
+        hook = options[:concurrency]&.on_item_finish_in_child_process
+        if hook && !assignment["dry_run"]
+          begin
+            hook.call({test_run_id: assignment.fetch("test_run_id"), item: Replay.public_replay_items([result]).first})
+          rescue => error
+            stderr.puts "[replay] child grading hook failed for #{result[:original_trace_id]} attempt #{result[:attempt]}: #{error}"
+          end
         end
+        unless client.close(timeout: delivery_timeout) || assignment["dry_run"]
+          Bitfab.count_delivery_problem("delivery was not confirmed within child_delivery_timeout (#{delivery_timeout}s). " \
+            "Spans that failed or were still sending when the child exited may never reach Bitfab.")
+        end
+        write_delivery_problems(assignment["delivery_problems_path"], result, Bitfab.delivery_problem_counts, stderr)
       end
-      client.close(timeout: 5)
       result
     end
 
+    def write_delivery_problems(path, result, problems, stderr)
+      return if problems.empty?
+      label = "#{result[:original_trace_id]}##{result[:attempt]}"
+      report = problems.map do |message, count|
+        "[replay] span delivery problem for #{label}: #{message}#{" (x#{count})" if count > 1}\n"
+      end.join
+      stderr.write(report)
+      File.write(path, report, mode: "w", perm: 0o600) if path
+    rescue SystemCallError
+      nil
+    end
+
     def write_result(path, result)
       encoded = begin
         Marshal.dump(result)
@@ -116,7 +136,8 @@ module Bitfab
         result_path = File.join(@directory, "#{index}.result")
         assignment_path = File.join(@directory, "#{index}.json")
         stdout_path, stderr_path = File.join(@directory, "#{index}.out"), File.join(@directory, "#{index}.err")
-        assignment = {pipeline: @args.fetch(:pipeline), params: @params, item:, test_run_id:, mock:, include_db_branch_lease:, db_branch_settings:, dry_run:, result_path:}
+        delivery_problems_path = File.join(@directory, "#{index}.delivery")
+        assignment = {pipeline: @args.fetch(:pipeline), params: @params, item:, test_run_id:, mock:, include_db_branch_lease:, db_branch_settings:, dry_run:, result_path:, delivery_problems_path:}
         File.write(assignment_path, JSON.generate(assignment), mode: "w", perm: 0o600)
         executable = File.expand_path("../../exe/bitfab-replay", __dir__)
         pid = @mutex.synchronize do
@@ -154,6 +175,8 @@ module Bitfab
         end
         child_errors = read_tail(stderr_path)
         @mutex.synchronize { @stderr.write(child_errors) } unless child_errors.empty?
+        delivery_problems = File.file?(delivery_problems_path) ? File.read(delivery_problems_path) : ""
+        @mutex.synchronize { @stderr.write(delivery_problems) } unless delivery_problems.empty? || child_errors.include?(delivery_problems)
         unless File.file?(result_path)
           output_tail = read_tail(stdout_path)
           @mutex.synchronize { @stderr.write(output_tail) } unless output_tail.empty?

data/lib/bitfab/replay_registry.rb

@@ -220,7 +220,8 @@ module Bitfab
                            end,
           memory_throttle: args.fetch(:memory_throttle, configured ? configured.memory_throttle : ReplayConcurrency::UNSET),
           on_item_finish_in_child_process: configured&.on_item_finish_in_child_process,
-          child_timeout: configured&.child_timeout || 2400
+          child_timeout: configured&.child_timeout || 2400,
+          child_delivery_timeout: configured&.child_delivery_timeout || ReplayConcurrency::UNSET
         )
         ReplayProcesses.current&.configure(registry, options[:concurrency]) if options[:concurrency].primitive == "process"
       end

data/lib/bitfab/simulation_plan.rb

@@ -2,7 +2,7 @@
 
 module Bitfab
   class SimulationPlan
-    CONTENT_KEYS = %w[input input_meta output output_meta input_serialized output_serialized].freeze
+    CONTENT_KEYS = %w[input input_meta output output_meta input_serialized output_serialized prompt].freeze
     FRAMEWORKS = %w[openai-agents langgraph claude-agent-sdk vercel-ai].freeze
     READ_TIMEOUT = 5.0
     REFRESH_SECONDS = 60.0
@@ -54,6 +54,7 @@ module Bitfab
       @draining = Set.new
       @stopped = false
       @first_read_finished = false
+      @first_read_deadline = nil
       self.class.register(self)
     end
 
@@ -144,14 +145,13 @@ module Bitfab
       @mutex.synchronize { @content_off.nil? && !reading? }
     end
 
-    def wait_for_first_read(timeout = READ_TIMEOUT)
-      reset_after_fork
+    def wait_for_first_read
+      refresh
       return if disabled?
 
-      deadline = now + [timeout, 0].max
       @mutex.synchronize do
-        while reading? && @thread
-          remaining = deadline - now
+        while reading? && @thread && @first_read_deadline
+          remaining = @first_read_deadline - now
           break unless remaining.positive?
           @condition.wait(@mutex, remaining)
         end
@@ -229,6 +229,7 @@ module Bitfab
     end
 
     def start_reader(delay)
+      @first_read_deadline ||= now + delay + READ_TIMEOUT
       @thread = Thread.new { read_until_loaded(delay) }
       @thread.name = "bitfab-sim-plan"
     rescue ThreadError
@@ -302,6 +303,7 @@ module Bitfab
       @thread = nil
       @held = []
       @draining = Set.new
+      @first_read_deadline = nil unless @first_read_finished
     end
   end
 end

data/lib/bitfab/span_context.rb

@@ -150,10 +150,11 @@ module Bitfab
 
     # Execute a block with a new span pushed onto the stack.
     # The span is automatically popped when the block completes.
-    def with_span(trace_id:, span_id:, explicit_span_receiver: nil, explicit_span_method_name: nil, surface: nil, recording_override: nil)
+    def with_span(trace_id:, span_id:, explicit_span_receiver: nil, explicit_span_method_name: nil, surface: nil, recording_override: nil, content_withheld: nil)
       entry = {trace_id:, span_id:}
       entry[:surface] = surface if surface
       entry[:recording_override] = recording_override if recording_override
+      entry[:content_withheld] = content_withheld if content_withheld
       if explicit_span_receiver
         entry[:explicit_span_receiver] = explicit_span_receiver
         entry[:explicit_span_method_name] = explicit_span_method_name

data/lib/bitfab/subtree.rb

@@ -10,8 +10,9 @@ module Bitfab
 
   module Subtree
     DEFAULT_MAX_DEPTH = 30
-    DEFAULT_MAX_SPANS = 2048
-    DEFAULT_MAX_CAPTURED_SUBTREE_SPANS = 512
+    DEFAULT_MAX_SPANS = 10_000
+    DEFAULT_MAX_CAPTURED_SUBTREE_SPANS = 500
+    LIMIT_ORDER = %i[max_depth max_spans max_captured_subtree_spans].freeze
     SESSION_KEY = :bitfab_subtree_session
     DISPATCHER_KEY = :bitfab_subtree_dispatcher
     FIBER_CONTEXT_KEY = :bitfab_subtree_fiber_context
@@ -444,7 +445,8 @@ module Bitfab
         @method_metadata = Hash.new { |owners, owner| owners[owner] = {} }
         @spans_used = 1
         @captured_subtree_count = 0
-        @truncated = false
+        @limits_fired = Set.new
+        @dropped_spans = 0
         @exhausted = false
         @drained = false
         @deferred_frames = []
@@ -478,7 +480,7 @@ module Bitfab
         unless @deferred
           @fiber_parents.clear
           finish_open_spans
-          warn_if_truncated
+          record_truncation
         end
       end
 
@@ -491,7 +493,7 @@ module Bitfab
       end
 
       def handle(event)
-        return if @drained
+        return count_drained_call(event) if @drained
 
         if event.event == :raise
           record_error(event)
@@ -523,16 +525,21 @@ module Bitfab
       def reserve_node
         stack = current_stack(Fiber.current.object_id)
         if captured_depth(stack) >= @max_depth
-          @truncated = true
+          drop(:max_depth)
           return false
         end
         return true if take_slot(discovered: false)
 
-        @truncated = true
+        drop(:max_spans)
         @exhausted = true
         false
       end
 
+      def current_span_content_withheld?
+        entry = @stacks[Fiber.current.object_id]&.reverse_each&.find { |candidate| candidate.is_a?(OpenSpan) }
+        entry&.span_data&.dig("content_off_by_simulation_plan") == true
+      end
+
       def configured_entry(name:, type:, args:, kwargs:, test_run_id:)
         return unless reserve_node
 
@@ -624,11 +631,13 @@ module Bitfab
         fiber_id = Fiber.current.object_id
         if @suppressed_depths[fiber_id].to_i.positive?
           @suppressed_depths[fiber_id] += 1
+          count_dropped_call(event, resolved_path)
           return
         end
 
         if @exhausted
           suppress(fiber_id)
+          count_dropped_call(event, resolved_path)
           return
         end
 
@@ -638,13 +647,13 @@ module Bitfab
 
         stack = current_stack(fiber_id)
         if captured_depth(stack) >= @max_depth
-          @truncated = true
+          drop(:max_depth)
           suppress(fiber_id)
           return
         end
         content_off = content_off?(metadata.name)
         if (limit = limit_reached(discovered: true, content_off:))
-          @truncated = true
+          drop(limit)
           stop_discovery(fiber_id) unless limit == :max_captured_subtree_spans && content_off_names?
           return
         end
@@ -927,9 +936,35 @@ module Bitfab
         false
       end
 
-      def warn_if_truncated
-        return unless @truncated
+      def drop(limit)
+        @limits_fired << limit
+        @dropped_spans += 1
+      end
+
+      def count_dropped_call(event, resolved_path)
+        metadata = metadata_for(event, resolved_path)
+        return if metadata.excluded || (!@include_wrappers && metadata.passthrough)
+
+        @dropped_spans += 1
+      end
+
+      def count_drained_call(event)
+        return unless event.event == :call
+        return if skip_boundary_call?(event)
 
+        resolved_path = owned_path(event.path)
+        count_dropped_call(event, resolved_path) if resolved_path
+      rescue Exception # rubocop:disable Lint/RescueException
+        nil
+      end
+
+      def record_truncation
+        return if @limits_fired.empty?
+
+        CurrentTrace.new(@trace_id).set_metadata(
+          "bitfab.truncated_by" => LIMIT_ORDER.select { |limit| @limits_fired.include?(limit) }.join(","),
+          "bitfab.dropped_spans" => @dropped_spans
+        )
         Bitfab.warn_once(
           "subtree-truncated:#{@trace_function_key}",
           "'#{@trace_function_key}' hit a subtree capture limit (max_depth=#{@max_depth}, " \

data/lib/bitfab/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Bitfab
-  VERSION = "0.57.6"
+  VERSION = "0.57.8"
 end

data/lib/bitfab/warn_once.rb

@@ -12,6 +12,8 @@ module Bitfab
   # warns once, not just the first one seen.
   @warn_once_seen = Set.new
   @warn_once_mutex = Mutex.new
+  @delivery_problems = nil
+  @delivery_problems_mutex = Mutex.new
 
   class << self
     def warn_once(key, message)
@@ -37,6 +39,27 @@ module Bitfab
       # Logging must never crash the host app (e.g. a closed $stderr).
     end
 
+    def warn_delivery_problem(message)
+      count_delivery_problem(message)
+      warn_always(message)
+    end
+
+    def count_delivery_problem(message)
+      @delivery_problems_mutex.synchronize { @delivery_problems[message] += 1 if @delivery_problems }
+    end
+
+    def delivery_problem_counts
+      @delivery_problems_mutex.synchronize { @delivery_problems ? @delivery_problems.dup : {} }
+    end
+
+    def collect_delivery_problems
+      problems = Hash.new(0)
+      @delivery_problems_mutex.synchronize { @delivery_problems = problems }
+      yield problems
+    ensure
+      @delivery_problems_mutex.synchronize { @delivery_problems = nil }
+    end
+
     # Test-only: clear the dedup set so a warning can fire again.
     def _reset_warn_once
       @warn_once_mutex.synchronize { @warn_once_seen.clear }

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: bitfab
 version: !ruby/object:Gem::Version
-  version: 0.57.6
+  version: 0.57.8
 platform: ruby
 authors:
 - Harvest Team
@@ -161,6 +161,7 @@ files:
 - lib/bitfab/client.rb
 - lib/bitfab/commit_ref.rb
 - lib/bitfab/compress.rb
+- lib/bitfab/connection_pool.rb
 - lib/bitfab/constants.rb
 - lib/bitfab/datasets.rb
 - lib/bitfab/db_snapshot.rb