rails_mind 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fdc13f69dfd22c26fb7bf8d6f8cbf3f202d0cd7e5fd118d84611473f6486c2ab
-  data.tar.gz: af85f41317d417ad0521d2c93258ebee6f0605c6fbdfb0f0d3bbbabf759555a6
+  metadata.gz: 819a570e3f4f65e0fb8bae7de68b1a81ee744bccb77349858c640fcc912e95b4
+  data.tar.gz: 62df4fca70603e28458d277dcd8d734a5de04af7c514f997e15686122a02dbd2
 SHA512:
-  metadata.gz: a6c3fa13f2cbf791b1e6d0128e23d2060454386f39bb3ceb97817c368f1a4577145bae249d15db009cf8b967a65474e767dedcf97710af3b238bdde3334a12f9
-  data.tar.gz: 6a8eee210906bde6ab8edda408966e535139ba3ce114eda1a91ff3075700223e2dde8f867dd65727f14f80be38da0c7c8783d6d905adf513d5afcb20f746c4d3
+  metadata.gz: 3583084684b6ac4b165864c14fbfe3922fc2f678c8d7aa9383418d95fdc528083bbe2327e5428d167d80e54a28445309eb6243bd3c6c3dac36fc8a67fa9e0630
+  data.tar.gz: e968233e4025ce470a83207214ef886032bff662faea00f824f3aceefad55319aa6d3235b62cc16368c5b8ccec7db86a6914ce1d5cb1a5db2c9bb282a543f0f2

data/CHANGELOG.md

@@ -1,5 +1,12 @@
 # Changelog
 
+## 0.3.0 — 2026-10-10
+
+- Enable bounded automatic endpoint sampling for routine successful requests, driven by expiring hosted policies. Preserve slow, failed, repeated-query requests and all other signal kinds from SDK sampling.
+- Report eligible request selection and queue losses with immutable IDs and independent acknowledgments, including report-only batches at zero sampling rates. Keep old-server event delivery compatible until a v1 handshake.
+- Add explicit local opt-out, effective request metadata, bounded control parsing, fork resets, and sampling diagnostics. Local opt-out does not disable server-side sampling or quotas.
+- Separate known local drops from unconfirmed delivery; failed responses may follow partial server acceptance.
+
 ## 0.2.0 — 2026-10-09
 
 - Correlate requests, serialized ActiveJob executions, and mailer attempts with native trace IDs while preserving existing OpenTelemetry context.

data/README.md

@@ -2,11 +2,11 @@
 
 The MIT-licensed Ruby client for [RailsMind](https://railsmind.com). One bounded
 collector for Rails requests, errors, jobs, mailer attempts, Ahoy events, Flipper
-observations, and selected OpenTelemetry spans. Version 0.2.0 is an early release; see
+observations, and selected OpenTelemetry spans. Version 0.3.0 is an early release; see
 [compatibility and validation limits](docs/compatibility.md).
 
 ```ruby
-gem "rails_mind", "~> 0.2.0"
+gem "rails_mind", "~> 0.3.0"
 ```
 
 Install from RubyGems.org and commit your application's `Gemfile.lock` to pin the
@@ -67,4 +67,4 @@ bundle exec ruby script/package_smoke.rb
 ruby script_benchmark.rb
 ```
 
-Business/request/error events are unsampled, but delivery is best effort. Check `RailsMind.stats` for losses. The SDK does not guarantee durable business-event counts; use a transactional outbox when that is required.
+Automatic endpoint sampling is enabled by default. The hosted policy can reduce routine successful requests while preserving failed, slow, or repeated-query requests and all other signal kinds from sampling. Business/error events remain unsampled; bounded queues and delivery are still best effort. See [sampling and coverage](docs/sdk.md#automatic-endpoint-sampling) and inspect `RailsMind.stats`. Use a transactional outbox when durable business-event delivery is required.

data/docs/compatibility.md

@@ -94,3 +94,19 @@ recipient inbox delivery, direct backend jobs, direct provider HTTP sends, and
 durable workflow completeness are outside this contract. Validate the released
 SDK against your app's queue, callback ordering, and delivery adapter. The hosted
 service exposes bounded workflow metadata and leaves recipient delivery unknown.
+
+## 0.3.0 endpoint sampling verification
+
+The Ruby suite passes locally with Ruby 4.0.7 and the locked Rails 8.1.4
+libraries: 105 tests and 1,311 assertions, with no failures or skips. The three
+Node browser-adapter contract tests also pass. These are isolated SDK/library
+tests; they do not boot a Rails application or use a database. CI separately
+runs the packaged-gem installation smoke check on Ruby 4.0.6.
+
+Coverage includes protected signals, deterministic shared selection, endpoint
+rates, local opt-out, malformed/expired controls, monotonic expiry, queue loss,
+immutable reports and acknowledgments, quota-response retries, old-server
+fallback, idle zero-rate report refresh, streamed response bounds, batch byte
+limits, and real process fork resets. Fake transports verify the wire contract;
+deployed server policy behavior, cross-process traffic forecasts, and a
+customer app's production overhead require hosted/staging validation.

data/docs/sdk.md

@@ -3,13 +3,13 @@
 The gem is `rails_mind`, namespace `RailsMind`. Install the MIT-licensed client
 from RubyGems.org and commit the application lockfile to retain the selected
 version. The hosted RailsMind service is a separate, proprietary application.
-Version 0.2.0 is an early release with the [documented compatibility limits](compatibility.md).
+Version 0.3.0 is an early release with the [documented compatibility limits](compatibility.md).
 
 ## Install
 
 ```ruby
 # Customer Gemfile
-gem "rails_mind", "~> 0.2.0"
+gem "rails_mind", "~> 0.3.0"
 ```
 
 ```sh
@@ -180,9 +180,74 @@ RailsMind::Integrations::OpenTelemetry.attach!
 
 Attachment adds one idempotent processor; it does not replace the global provider, sampler, propagator, instrumentation, or other exporters. Its exporter only sanitizes and enqueues locally; the shared worker performs network I/O. Span context is attached at start so identity survives later export. Known safe attributes are allowlisted. Raw span names, exception span events, HTTP bodies/headers, SQL statements, and bind values are not exported to RailsMind. Existing third-party exporters retain their own behavior and privacy settings. [OpenTelemetry exporters](https://opentelemetry.io/docs/languages/ruby/exporters/).
 
-Request durations/counts and error reports come independently from Rails notifications/error reporting, even when detailed traces are sampled out. Request measurements cover controller actions, not assets/health checks rejected before ActionController or streaming-body completion. Repeated-query evidence contains only normalized-query hashes and counts; it is a suspicion, not proof of an N+1. Hash normalization handles common literals, not every SQL grammar. Use Bullet/Prosopite or an explicit query-bound test in development to verify fixes. No production object scans are installed.
+Request measurements and error reports come independently from Rails notifications/error reporting, even when detailed traces are sampled out. Routine requests separately follow the automatic endpoint sampling policy described below. Request measurements cover controller actions, not assets/health checks rejected before ActionController or streaming-body completion. Repeated-query evidence contains only normalized-query hashes and counts; it is a suspicion, not proof of an N+1. Hash normalization handles common literals, not every SQL grammar. Use Bullet/Prosopite or an explicit query-bound test in development to verify fixes. No production object scans are installed.
 
-Full metrics/logs SDK integration is deferred; hosted aggregates use unsampled request events and the SDK does not capture application logs.
+Full metrics/logs SDK integration is deferred. Hosted request aggregates describe retained events; sampling can bias their distributions. The SDK does not capture application logs.
+
+## Automatic endpoint sampling
+
+Version 0.3.0 enables automatic endpoint sampling by default. A compatible hosted
+service supplies environment-wide endpoint rates based on its allowance policy.
+The SDK applies them before queueing, reducing serialization, network, and retained
+event volume. Each process uses the same policy and deterministic event-ID hash;
+there are no independent per-process allowance grants. A server-side tightening
+uses the same selection rule, so the two samplers do not multiply probabilities.
+
+Only completed `request` events with status 200–399, a finite duration from zero
+to below 1,000 ms, and fewer than three repeated queries are eligible. Slow,
+failed, unknown-duration, and repeated-query requests remain unsampled, as do
+business events, errors, jobs, mailer observations, visits, flags, checks, and
+OpenTelemetry spans. This preserves their sampling priority, not guaranteed
+delivery: queue bounds, payload limits, outages, and the server's hard daily quota
+still apply. Existing OpenTelemetry samplers remain authoritative; this feature
+does not promise complete distributed traces.
+
+To turn off SDK request selection explicitly:
+
+```ruby
+RailsMind.configure do |config|
+  config.automatic_sampling = false
+end
+```
+
+The SDK communicates this choice on policy and ingestion requests. The
+environment owner's hosted setting separately controls server-side sampling;
+local opt-out does **not** disable it or bypass the daily quota.
+
+The background worker requests `/api/v1/sampling_policy` with the ingest key and
+refreshes at most once per minute after its first request observation, including
+when the current rate is zero. It also receives controls on successful ingestion
+and quota responses. Missing, malformed, or expired policies fall back to rate 1;
+this may exhaust the server allowance during an outage. Expiry uses wall and
+monotonic clocks, with a maximum accepted lifetime of ten minutes. Before a valid
+v1 handshake, old servers and custom transports receive ordinary event-only
+batches and retain normal flush behavior. Forked workers reset queues, policies,
+counters, and report IDs and fetch their own copy of the shared policy.
+
+Retained request properties include SDK-owned `rails_mind_sampling` metadata:
+`version`, `policy_id`, effective `rate`, and `state` (`active`, `unavailable`,
+`expired`, `disabled`, or `protected`). This is the SDK selection decision, not
+proof of server acceptance or a complete traffic denominator.
+
+At most once per minute per process, the worker freezes an identified report of
+routine eligible observations; an explicit flush may send it sooner. Reports
+contain controller/action names and counts only: `observed`, `retained` (selected
+for enqueue), `sampled_out`, and `queue_dropped` (a subset of retained). They never
+contain event properties or user/account identifiers. Counts are observed after
+validation, redaction, and `before_send`; hooks that suppress events, crashes,
+protected requests, and unknown endpoint names are outside this denominator.
+Retained counts are not accepted counts. The resulting request percentiles and
+rates cannot be treated as an unbiased whole-traffic distribution.
+
+Reports have immutable UUIDs/bodies and separate durable acknowledgments. They
+can travel in `events: []` batches without consuming event quota, and an
+acknowledged report does not establish delivery of its accompanying events.
+Counters cover at most 100 endpoint names per window, reports share the normal
+batch byte limit, and each process retains at most five unacknowledged reports.
+Overflows or invalid clock windows mark coverage `limited`; reports older than
+seven days are discarded. This is bounded in-memory observation, not a durable
+request ledger. Unacknowledged reports after a confirmed handshake can cause
+`RailsMind.flush` to return false at its deadline.
 
 ## Jobs and mailer workflows
 
@@ -250,16 +315,18 @@ Add `<meta name="rails-mind-page" content="Invoices#index">` to supply a stable
 
 ## Bounds, privacy, and outages
 
-Defaults are one lazy worker per process, 1,000 queued/in-flight events, 16 KiB per event, at most 100 events and 256 KiB per batch, and a two-second flush interval. Overflow drops the newest event. Retries retain the same encoded bytes/UUIDs; only HTTP 429, 5xx, and transient transport failures retry, at most three retries with bounded backoff. Other 4xx responses drop the batch. Redirects are not followed. Shutdown gets two seconds by default, then discards outstanding telemetry; forks start a fresh child queue rather than resending the parent's buffer.
+Defaults are one lazy worker per process, 1,000 queued/in-flight events, 16 KiB per event, at most 100 events and 256 KiB per batch, and a two-second flush interval. Overflow drops the newest event. Retries retain the same encoded bytes/UUIDs; only HTTP 429, 5xx, and transient transport failures retry, at most three retries with bounded backoff. Other 4xx responses end delivery attempts. A failed response can follow partial server acceptance, so such events are classified as delivery unconfirmed. Redirects are not followed. Shutdown gets two seconds by default, then discards outstanding telemetry; forks start a fresh child queue rather than resending the parent's buffer.
 
 ```ruby
 RailsMind.stats
-# => { enqueued:, delivered:, dropped:, retried:, queued:, in_flight:,
-#      losses: { "event.overflow" => 3, "error.delivery" => 1 } }
+# => { enqueued:, delivered:, dropped:, delivery_unconfirmed:, retried:, queued:, in_flight:,
+#      losses: { "event.overflow" => 3, "error.delivery_unconfirmed" => 1 },
+#      sampling: { observed:, retained:, sampled_out:, queue_dropped:,
+#                  unreported:, reports_lost:, pending_reports:, limited:, policy_state:, policy_id: } }
 RailsMind.flush(timeout: 2)
 ```
 
-These are per-process counters. The doctor prints them for its process; inspect running app stats through your existing operational tooling. The hosted app does not currently receive loss counters. Request/business/error signals are unsampled, but process crashes, rejected payloads, redaction hooks, exhausted retries, or queue pressure can still undercount. This is best-effort monitoring, not an accounting ledger. A future transactional outbox can provide durable business-event delivery; the current implementation does not claim that guarantee.
+These are per-process counters. `dropped` counts known local discards, such as queue overflow or oversize payloads. `delivery_unconfirmed` counts events whose delivery attempts ended without a success response, including interrupted in-flight batches; some may have been accepted by the server. `delivered` counts submitted events in batches with a successful HTTP response; it does not prove server retention or background processing. The doctor prints stats for its process; inspect running app stats through your existing operational tooling. Only the routine-request report above sends queue-loss observations to the hosted app. All signal kinds can still undercount after crashes, rejected payloads, hooks, or outages. This is best-effort monitoring, not an accounting ledger; use a transactional outbox when durable business-event delivery is required.
 
 Redaction occurs before buffering and again after custom `before_send`. Sensitive property names are filtered, nesting/string/collection sizes are capped, emails and common credential forms are scrubbed, exception messages are omitted by default, and backtraces drop machine-home prefixes. Bodies, cookies, credentials, SQL binds, and job arguments are never selected by integrations. Arbitrary user-provided text may contain sensitive content that generic redaction cannot recognize: use small, explicitly reviewed business properties. Opting into exception messages requires reviewing `capture_exception_message` and your custom scrubbing policy.
 

data/lib/generators/rails_mind/install/templates/initializer.rb.tt

@@ -1,4 +1,7 @@
 RailsMind.configure do |config|
+  # Enabled by default: reduce routine successful request events using hosted policy.
+  # Local opt-out does not disable server sampling or the environment daily quota.
+  # config.automatic_sampling = false
   config.token = ENV["RAILS_MIND_KEY"] # Environment-scoped server credential. Never expose to browsers.
 
   # Endpoint defaults to https://railsmind.com; RAILS_MIND_ENDPOINT overrides it.

data/lib/rails_mind/collector.rb

@@ -7,17 +7,28 @@ module RailsMind
 
     def push(event)
       check_process
-      json = JSON.generate(event)
+      event = event.transform_keys(&:to_s)
       @mutex.synchronize do
-        reason = if @closed then :closed
-        elsif json.bytesize > @config.max_event_bytes || json.bytesize + 13 > @config.max_batch_bytes then :oversize
+        if @closed
+          drop(event["kind"], :closed)
+          return false
+        end
+        selected, observation = @sampling.consider(event)
+        unless selected
+          start_worker if @autostart
+          return false
+        end
+        json = JSON.generate(event)
+        reason = if json.bytesize > @config.max_event_bytes || json.bytesize + 13 > @config.max_batch_bytes then :oversize
         elsif @queue.size + @in_flight >= @config.queue_capacity then :overflow
         end
         if reason
-          drop(event[:kind] || event["kind"], reason)
+          @sampling.queue_dropped(observation)
+          drop(event["kind"], reason)
+          start_worker if @autostart
           return false
         end
-        @queue << [ json.freeze, event[:kind] || event["kind"] ]
+        @queue << [ json.freeze, event["kind"] ]
         @stats[:enqueued] += 1
         start_worker if @autostart
         @condition.broadcast if @queue.size >= @config.batch_size
@@ -27,7 +38,7 @@ module RailsMind
 
     def stats
       check_process
-      @mutex.synchronize { @stats.merge(queued: @queue.size, in_flight: @in_flight, losses: @losses.dup) }
+      @mutex.synchronize { @stats.merge(queued: @queue.size, in_flight: @in_flight, losses: @losses.dup, sampling: @sampling.stats) }
     end
 
     def flush(timeout: @config.shutdown_timeout)
@@ -35,9 +46,9 @@ module RailsMind
       deadline = monotonic + timeout
       @mutex.synchronize do
         start_worker
-        @force_flush = true
+        @force_flush = @force_report = true
         @condition.broadcast
-        while @queue.any? || @in_flight.positive?
+        while @queue.any? || @in_flight.positive? || @active_report || @sampling.pending?
           remaining = deadline - monotonic
           return false if remaining <= 0
           @condition.wait(@mutex, remaining)
@@ -58,9 +69,10 @@ module RailsMind
         @mutex.synchronize do
           @queue.each { |_, kind| drop(kind, :shutdown) }
           @queue.clear
-          @active_batch.to_a.each { |_, kind| drop(kind, :shutdown) }
-          @active_batch = nil
+          @active_batch.to_a.each { |_, kind| unconfirmed(kind) }
+          @active_batch = @active_report = nil
           @in_flight = 0
+          @sampling.discard!
         end
       end
       success
@@ -73,13 +85,15 @@ module RailsMind
       @mutex = Mutex.new
       @condition = ConditionVariable.new
       @queue, @losses = [], Hash.new(0)
-      @stats = { enqueued: 0, delivered: 0, dropped: 0, retried: 0 }
+      @stats = { enqueued: 0, delivered: 0, dropped: 0, delivery_unconfirmed: 0, retried: 0 }
+      @sampling = Sampling.new(@config)
       @in_flight = 0
-      @worker = @active_batch = nil
-      @closed = @stopping = @force_flush = false
+      @worker = @active_batch = @active_report = nil
+      @closed = @stopping = @force_flush = @force_report = false
     end
 
-    # Never resend a parent's inherited queue after Puma/worker forks.
+    # No inherited observations, report IDs, policies or locked mutexes survive
+    # a Puma/worker fork. Each child refreshes the same environment-wide policy.
     def check_process
       reset_process if @pid != Process.pid
     end
@@ -94,46 +108,73 @@ module RailsMind
 
     def run
       loop do
-        batch = @mutex.synchronize do
+        refresh = @mutex.synchronize do
           @condition.wait(@mutex, @config.flush_interval) if @queue.size < @config.batch_size && !@force_flush && !@stopping
           return if @stopping && @queue.empty?
-          take_batch
+          due = @sampling.policy_due? && @transport.respond_to?(:fetch_sampling_policy)
+          @sampling.policy_requested! if due
+          due
         end
-        next if batch.empty?
-        result = deliver(batch)
+        refresh_policy if refresh
+        batch, body = @mutex.synchronize { take_batch }
+        next unless body
+        result = deliver(body, report_id: @active_report)
         @mutex.synchronize do
           if result
             @stats[:delivered] += batch.size
           else
-            batch.each { |_, kind| drop(kind, :delivery) }
+            batch.each { |_, kind| unconfirmed(kind) }
           end
           @in_flight = 0
-          @active_batch = nil
-          @force_flush = false if @queue.empty?
+          @active_batch = @active_report = nil
+          @force_flush = false if @queue.empty? && !@sampling.pending?
           @condition.broadcast
         end
       end
     end
 
     def take_batch
+      @sampling.rotate(force: @force_report, byte_budget: @config.max_batch_bytes - 64)
+      report = @sampling.report(force: @force_report)
+      @force_report = false if @sampling.supported?
       batch = []
-      bytes = 13
+      report_json = report && JSON.generate(report)
+      bytes = 13 + (report_json ? 19 + report_json.bytesize : 0)
       while batch.size < @config.batch_size && @queue.any?
         entry = @queue.first
         break if bytes + entry.first.bytesize + (batch.empty? ? 0 : 1) > @config.max_batch_bytes
         bytes += entry.first.bytesize + (batch.empty? ? 0 : 1)
         batch << @queue.shift
       end
+      if batch.empty? && !report
+        @force_flush = false
+        @condition.broadcast
+        return [ [], nil ]
+      end
       @in_flight = batch.size
       @active_batch = batch
-      batch
+      @active_report = report && report["id"]
+      @sampling.report_sent! if report
+      body = '{"events":[' + batch.map(&:first).join(",") + ']'
+      body += ',"sampling_report":' + report_json if report_json
+      [ batch, (body + '}').freeze ]
     end
 
-    def deliver(batch)
-      body = '{"events":[' + batch.map(&:first).join(",") + ']}'
+    def refresh_policy
+      result = @transport.fetch_sampling_policy
+      @mutex.synchronize { @sampling.apply_policy(result.sampling_policy) }
+    rescue StandardError
+      nil # Missing/expired controls fall back to unsampled collection.
+    end
+
+    def deliver(body, report_id:)
       attempts = 0
       loop do
         result = @transport.send_batch(body)
+        @mutex.synchronize do
+          @sampling.apply_policy(result.sampling_policy) if result.respond_to?(:sampling_policy)
+          @sampling.acknowledge(result.sampling_report_ack) if report_id && result.respond_to?(:sampling_report_ack) && result.sampling_report_ack == report_id
+        end
         return true if result.success
         return false unless result.retryable && attempts < @config.max_retries
         delay = result.retry_after || @config.retry_base * (2**attempts) * (0.75 + rand * 0.5)
@@ -142,7 +183,12 @@ module RailsMind
         sleep(delay) if delay.positive?
       end
     rescue StandardError
-      false # Telemetry errors cannot escape into the customer app.
+      false
+    end
+
+    def unconfirmed(kind)
+      @stats[:delivery_unconfirmed] += 1
+      @losses["#{kind}.delivery_unconfirmed"] += 1
     end
 
     def drop(kind, reason)

data/lib/rails_mind/configuration.rb

@@ -9,7 +9,7 @@ module RailsMind
     attr_accessor :endpoint, :token, :release, :apm, :errors, :analytics, :flags,
       :queue_capacity, :batch_size, :max_batch_bytes, :max_event_bytes, :flush_interval,
       :max_retries, :retry_base, :open_timeout, :read_timeout, :shutdown_timeout,
-      :before_send, :capture_exception_message
+      :before_send, :capture_exception_message, :automatic_sampling
 
     def initialize(env: ENV, root: nil)
       @endpoint = nonblank(env["RAILS_MIND_ENDPOINT"]) || DEFAULT_ENDPOINT
@@ -27,12 +27,14 @@ module RailsMind
       @read_timeout = 2.0
       @shutdown_timeout = 2.0
       @capture_exception_message = false
+      @automatic_sampling = true
     end
 
     def enabled? = !token.to_s.strip.empty? && !endpoint.to_s.strip.empty?
 
     def validate!
       return self unless enabled?
+      raise ArgumentError, "automatic_sampling must be true or false" unless automatic_sampling == true || automatic_sampling == false
       uri = URI(endpoint)
       loopback = %w[localhost 127.0.0.1 ::1].include?(uri.hostname)
       raise ArgumentError, "endpoint must use HTTPS (HTTP only on loopback)" unless uri.is_a?(URI::HTTPS) || (uri.is_a?(URI::HTTP) && loopback)

data/lib/rails_mind/diagnostic.rb

@@ -21,15 +21,16 @@ module RailsMind
         endpoint_present: !@config.endpoint.to_s.strip.empty?,
         release_present: !@config.release.to_s.strip.empty?,
         enabled: @config.enabled?,
+        automatic_sampling: @config.automatic_sampling,
         plan: [
           "Add one initializer. No database migrations or Gemfile changes.",
-          "Requests and jobs: unsampled measurements via Rails notifications; errors via Rails.error.",
+          "Requests: automatic sampling of routine successes; jobs via Rails notifications and errors via Rails.error stay unsampled.",
           detected["ahoy_matey"] ? "Ahoy detected: keep current store; explicitly opt into mirror mode after review." : "Analytics: optionally add ahoy_matey; choose hosted-only store or existing database store.",
           detected["opentelemetry-sdk"] ? "OpenTelemetry detected: attach RailsMind exporter after existing SDK setup; preserve sampler and exporters." : "APM traces: optionally add selected OpenTelemetry libraries and explicitly configure one provider.",
           detected["flipper"] ? "Flipper detected: observe notifications only; preserve adapter, groups, targeting and cache." : "Flags: Flipper is optional. No flag configuration is created.",
           "Existing error reporters stay installed. RailsMind deduplicates its own collector only.",
           "Browser: optional Ahoy adapter; enable only after checking existing pageview handlers.",
-          "Business/request/error signals are not sampled; bounded queue losses remain possible. Inspect RailsMind.stats."
+          "Business/error signals are not sampled; routine requests follow the hosted policy. Local opt-out does not disable server sampling or quotas. Inspect RailsMind.stats for queue losses and unconfirmed delivery."
         ],
         limitations: [ "Compatibility applies to tested versions only; this is not a historical import.",
           "No network check is performed and credential values are never printed." ]

data/lib/rails_mind/sampling.rb

@@ -0,0 +1,211 @@
+require "json"
+require "securerandom"
+require "time"
+require "digest"
+
+module RailsMind
+  # Called under Collector's mutex. Decisions and queue-loss accounting share
+  # the same critical section as report rotation; reports are immutable once cut.
+  class Sampling
+    REPORT_INTERVAL = 60
+    POLICY_INTERVAL = 60
+    MAX_POLICY_TTL = 600
+    MAX_ENDPOINTS = 100
+    MAX_PENDING = 5
+    MAX_COUNT = 1_000_000_000
+    MAX_REPORT_AGE = 7 * 86_400
+    ENDPOINT = /\A[A-Z][A-Za-z0-9_]*(?:::[A-Z][A-Za-z0-9_]*)*#[A-Za-z_][A-Za-z0-9_!?]*\z/
+    UUID = /\A[0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12}\z/i
+
+    def initialize(config, clock: -> { Time.now.utc }, monotonic: -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) })
+      @config, @clock, @monotonic = config, clock, monotonic
+      @counts, @reports = {}, []
+      @from = @clock.call
+      @last_rotation = @monotonic.call
+      @next_policy_at = @next_report_at = 0
+      @policy = nil
+      @activated = false
+      @limited = false
+      @policy_ids = []
+      @stats = { observed: 0, retained: 0, sampled_out: 0, queue_dropped: 0, unreported: 0, reports_lost: 0 }
+    end
+
+    def consider(event)
+      return [ true, nil ] unless event["kind"] == "request"
+      @activated = true
+      rate, state, policy_id = effective_policy
+      name = event["name"]
+      unless eligible?(event) && name.is_a?(String) && name.bytesize <= 200 && name.match?(ENDPOINT)
+        annotate(event, 1.0, "protected", policy_id)
+        return [ true, nil ]
+      end
+      if !@counts.key?(name) && @counts.size >= MAX_ENDPOINTS
+        @stats[:unreported] += 1
+        @limited = true
+        annotate(event, 1.0, "unavailable", policy_id)
+        return [ true, nil ]
+      end
+      count = @counts[name] ||= { "name" => name, "observed" => 0, "retained" => 0, "sampled_out" => 0, "queue_dropped" => 0 }
+      if count["observed"] >= MAX_COUNT
+        @stats[:unreported] += 1
+        @limited = true
+        annotate(event, 1.0, "unavailable", policy_id)
+        return [ true, nil ]
+      end
+      rate = @policy[:endpoints].fetch(name, rate) if state == "active"
+      selected = rate >= 1 || (rate.positive? && fraction(event["id"]) < rate)
+      count["observed"] += 1
+      count[selected ? "retained" : "sampled_out"] += 1
+      @stats[:observed] += 1
+      @stats[selected ? :retained : :sampled_out] += 1
+      @policy_ids << (policy_id || state) unless @policy_ids.include?(policy_id || state) || @policy_ids.size >= 2
+      annotate(event, rate, state, policy_id) if selected
+      [ selected, count ]
+    end
+
+    def queue_dropped(count)
+      return unless count
+      count["queue_dropped"] += 1
+      @stats[:queue_dropped] += 1
+    end
+
+    def apply_policy(value)
+      return false unless value.is_a?(Hash) && value["version"] == 1 && boolean?(value["enabled"])
+      id = value["id"]
+      return false unless id.is_a?(String) && id.bytesize.between?(1, 200) && !id.match?(/\p{Cc}/)
+      issued = parse_time(value["issued_at"])
+      expires = parse_time(value["expires_at"])
+      now = @clock.call
+      return false unless issued && expires && issued <= now + 300 && expires > now && expires > issued && expires - issued <= MAX_POLICY_TTL
+      return false if @policy && issued < @policy[:issued_at]
+      rate = value["default_rate"]
+      rows = value["endpoints"]
+      return false unless valid_rate?(rate) && rows.is_a?(Array) && rows.size <= MAX_ENDPOINTS
+      endpoints = {}
+      rows.each do |row|
+        return false unless row.is_a?(Hash) && row["name"].is_a?(String) && row["name"].bytesize.between?(1, 800) && row["name"].length <= 200 && !row["name"].match?(/\p{Cc}/) && valid_rate?(row["rate"])
+        return false if endpoints.key?(row["name"])
+        endpoints[row["name"]] = row["rate"].to_f
+      end
+      @policy = { id: id, issued_at: issued, expires_at: expires, deadline: @monotonic.call + [ expires - now, MAX_POLICY_TTL ].min,
+        enabled: value["enabled"], default_rate: rate.to_f, endpoints: endpoints }
+      true
+    rescue ArgumentError, EncodingError
+      false
+    end
+
+    def policy_due?
+      @activated && @monotonic.call >= @next_policy_at
+    end
+
+    def policy_requested!
+      @next_policy_at = @monotonic.call + POLICY_INTERVAL
+    end
+
+    def rotate(force: false, byte_budget:)
+      return unless supported?
+      now = @clock.call
+      while @reports.first && Time.iso8601(@reports.first.fetch("from")) < now - MAX_REPORT_AGE
+        @reports.shift
+        @stats[:reports_lost] += 1
+        @limited = true
+      end
+      return if @counts.empty? || (!force && @monotonic.call - @last_rotation < REPORT_INTERVAL)
+      if now < @from || now - @from > 86_400
+        @stats[:unreported] += @counts.values.sum { |entry| entry["observed"] }
+        @limited = true
+      else
+        rows = []
+        template = { "version" => 1, "limited" => @limited, "id" => SecureRandom.uuid, "from" => @from.iso8601(6), "to" => now.iso8601(6),
+          "enabled" => @config.automatic_sampling, "policy_id" => @policy_ids.size == 1 ? @policy_ids.first : "mixed", "endpoints" => rows }
+        @counts.each_value do |entry|
+          candidate = template.merge("endpoints" => rows + [ entry ])
+          if JSON.generate(candidate).bytesize > byte_budget
+            store_report(template.merge("endpoints" => rows)) if rows.any?
+            rows = []
+            template = template.merge("id" => SecureRandom.uuid)
+          end
+          if JSON.generate(template.merge("endpoints" => [ entry ])).bytesize <= byte_budget
+            rows << entry.dup
+          else
+            @stats[:unreported] += entry["observed"]
+            @limited = true
+          end
+        end
+        store_report(template.merge("endpoints" => rows)) if rows.any?
+      end
+      @counts, @policy_ids = {}, []
+      @from, @last_rotation = now, @monotonic.call
+    end
+
+    def report(force: false)
+      @reports.first if force || @monotonic.call >= @next_report_at
+    end
+
+    def report_sent!
+      @next_report_at = @monotonic.call + REPORT_INTERVAL
+    end
+
+    def acknowledge(id)
+      return unless id.is_a?(String) && id.match?(UUID)
+      removed = @reports.reject! { |report| report["id"] == id }
+      @next_report_at = 0 if removed && @reports.any?
+    end
+
+    def discard!
+      @stats[:reports_lost] += @reports.size
+      @stats[:unreported] += @counts.values.sum { |entry| entry["observed"] }
+      @reports, @counts = [], {}
+    end
+
+    def supported? = !@policy.nil?
+    def pending? = supported? && (@reports.any? || @counts.any?)
+    def stats = @stats.merge(limited: @limited, pending_reports: @reports.size, policy_state: effective_policy[1], policy_id: @policy&.fetch(:id))
+
+    private
+
+    def eligible?(event)
+      status, duration = event["status"], event["duration_ms"]
+      repeated = event.fetch("properties", {})["repeated_query_count"]
+      status.is_a?(String) && status.match?(/\A[23][0-9]{2}\z/) && duration.is_a?(Numeric) && duration.finite? && duration >= 0 && duration < 1000 && !(repeated.is_a?(Numeric) && repeated >= 3)
+    end
+
+    def effective_policy
+      return [ 1.0, "disabled", @policy&.fetch(:id) ] unless @config.automatic_sampling
+      return [ 1.0, "unavailable", nil ] unless @policy
+      return [ 1.0, "expired", @policy[:id] ] if @monotonic.call >= @policy[:deadline] || @clock.call >= @policy[:expires_at]
+      return [ 1.0, "disabled", @policy[:id] ] unless @policy[:enabled]
+      [ @policy[:default_rate], "active", @policy[:id] ]
+    end
+
+    def annotate(event, rate, state, policy_id)
+      event["properties"] = event.fetch("properties", {}).merge("rails_mind_sampling" => {
+        "version" => 1, "policy_id" => policy_id, "rate" => rate, "state" => state })
+    end
+
+    def store_report(report)
+      if @reports.size >= MAX_PENDING
+        @reports.shift
+        @stats[:reports_lost] += 1
+        @limited = true
+      end
+      report["limited"] ||= @limited
+      @reports << deep_freeze(JSON.parse(JSON.generate(report)))
+    end
+
+    def deep_freeze(value)
+      case value
+      when Hash then value.each { |key, child| deep_freeze(key); deep_freeze(child) }
+      when Array then value.each { |child| deep_freeze(child) }
+      end
+      value.freeze
+    end
+
+    def fraction(id) = Digest::SHA256.digest("rails_mind:sampling:v1:#{id.to_s.downcase}").unpack1("Q>") / (2.0**64)
+    def boolean?(value) = value == true || value == false
+    def valid_rate?(value) = (value.is_a?(Float) || value.is_a?(Integer)) && value.finite? && value.between?(0, 1)
+    def parse_time(value)
+      Time.iso8601(value) if value.is_a?(String) && value.bytesize <= 40
+    end
+  end
+end

data/lib/rails_mind/transport.rb

@@ -3,7 +3,8 @@ require "json"
 
 module RailsMind
   class Transport
-    Result = Struct.new(:success, :retryable, :retry_after, keyword_init: true)
+    MAX_RESPONSE_BYTES = 64 * 1024
+    Result = Struct.new(:success, :retryable, :retry_after, :sampling_policy, :sampling_report_ack, keyword_init: true)
     TRANSIENT_ERRORS = [ Timeout::Error, IOError, EOFError, SocketError, Errno::ECONNRESET,
       Errno::ECONNREFUSED, Errno::ETIMEDOUT, Errno::EPIPE, OpenSSL::SSL::SSLError ].freeze
 
@@ -15,28 +16,65 @@ module RailsMind
     end
 
     def send_batch(json)
+      request = Net::HTTP::Post.new(@uri)
+      request.body = json
+      perform(request)
+    end
+
+    def fetch_sampling_policy
+      uri = @uri.dup
+      uri.path = uri.path.sub(%r{/events$}, "/sampling_policy")
+      perform(Net::HTTP::Get.new(uri))
+    end
+
+    private
+
+    def perform(request)
       Context.suppress do
-        request = Net::HTTP::Post.new(@uri)
         request["Authorization"] = "Bearer #{@config.token}"
         request["Content-Type"] = "application/json"
         request["User-Agent"] = "rails_mind/#{VERSION}"
-        request.body = json
+        request["X-RailsMind-Automatic-Sampling"] = @config.automatic_sampling ? "enabled" : "disabled"
         http = @http_factory.call(@uri)
         http.use_ssl = @uri.scheme == "https"
         http.open_timeout = @config.open_timeout
         http.read_timeout = @config.read_timeout
         http.write_timeout = @config.read_timeout
         http.max_retries = 0
-        response = http.start { |connection| connection.request(request) }
+        body, streamed, oversized = +"", false, false
+        response = http.start do |connection|
+          connection.request(request) do |reply|
+            streamed = true
+            reply.read_body do |chunk|
+              if body.bytesize + chunk.bytesize > MAX_RESPONSE_BYTES
+                oversized = true
+              elsif !oversized
+                body << chunk
+              end
+            end
+          end
+        end
+        body = response.body.to_s if !streamed && response.respond_to?(:body)
         code = response.code.to_i
+        control = if !oversized && body.bytesize <= MAX_RESPONSE_BYTES && ((200..299).cover?(code) || code == 429)
+          parse_control(body)
+        else
+          {}
+        end
         Result.new(success: (200..299).cover?(code), retryable: code == 429 || code >= 500,
-          retry_after: retry_after(response["Retry-After"]))
+          retry_after: retry_after(response["Retry-After"]), sampling_policy: control["sampling_policy"],
+          sampling_report_ack: control["sampling_report_ack"])
       end
     rescue *TRANSIENT_ERRORS
       Result.new(success: false, retryable: true)
     end
 
-    private
+    def parse_control(body)
+      value = JSON.parse(body)
+      value.is_a?(Hash) ? value : {}
+    rescue JSON::ParserError, EncodingError
+      {}
+    end
 
     def retry_after(value)
       return nil unless value

data/lib/rails_mind/version.rb

@@ -1,3 +1,3 @@
 module RailsMind
-  VERSION = "0.2.0"
+  VERSION = "0.3.0"
 end

data/lib/rails_mind.rb

@@ -3,6 +3,7 @@ require_relative "rails_mind/configuration"
 require_relative "rails_mind/context"
 require_relative "rails_mind/redactor"
 require_relative "rails_mind/transport"
+require_relative "rails_mind/sampling"
 require_relative "rails_mind/collector"
 require_relative "rails_mind/client"
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rails_mind
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
 platform: ruby
 authors:
 - RailsMind
@@ -81,6 +81,7 @@ files:
 - lib/rails_mind/integrations/rails.rb
 - lib/rails_mind/railtie.rb
 - lib/rails_mind/redactor.rb
+- lib/rails_mind/sampling.rb
 - lib/rails_mind/transport.rb
 - lib/rails_mind/version.rb
 - lib/tasks/rails_mind.rake