rails_mind 0.2.0 → 0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fdc13f69dfd22c26fb7bf8d6f8cbf3f202d0cd7e5fd118d84611473f6486c2ab
-  data.tar.gz: af85f41317d417ad0521d2c93258ebee6f0605c6fbdfb0f0d3bbbabf759555a6
+  metadata.gz: 902c75e77dcb5d76b75de68e62e73fe7a92f38ec05f621084bc6a61c047cb186
+  data.tar.gz: b499762228b099f7d940a3b4aa58ba1d1c73d4a9c53958a02e379982fa08b28b
 SHA512:
-  metadata.gz: a6c3fa13f2cbf791b1e6d0128e23d2060454386f39bb3ceb97817c368f1a4577145bae249d15db009cf8b967a65474e767dedcf97710af3b238bdde3334a12f9
-  data.tar.gz: 6a8eee210906bde6ab8edda408966e535139ba3ce114eda1a91ff3075700223e2dde8f867dd65727f14f80be38da0c7c8783d6d905adf513d5afcb20f746c4d3
+  metadata.gz: ae733243113a0206980e2b34f2545f5648c00b4e032d32ae2727bd8b5da85d67067970cc73ff1654eef1eb93782457d71cb65c45e9a6bfcde5f6f7fc03985310
+  data.tar.gz: 48d196c72e82519469bc6afba1e47a585012a9ff3f0812c243ddd9b00378f6512f5b7d4f8d62fbd5158a029e2166d35e8d3373588fc3458c87c324848d477ed2

data/CHANGELOG.md

@@ -1,5 +1,21 @@
 # Changelog
 
+## 0.3.1 — unreleased
+
+- Send exception messages by default, scrubbed like every other string (emails, credentials, URL queries), and further for what generic scrubbing misses: database errors lose the failing row, key values, rejected input and quoted SQL literals, and printed hashes or keyword arguments lose the values of sensitive keys. `config.capture_exception_message = false` restores the placeholder. The flag is still not validated, so an app that assigned it from ENV boots as before.
+- Send the shape of each repeated query next to its fingerprint: the normalized SQL with literals replaced by `?`, comments removed, and at most 400 characters and 900 bytes, only for fingerprints that repeated within the request. One pass handles doubled and backslash-escaped quotes, `E''` and `$$` strings, hex numbers, comments containing quotes, a literal left open where a long statement is cut, and MySQL's double-quoted strings. `config.capture_query_shapes = false` sends fingerprints only.
+- Strip SQL comments (query log tags, Marginalia) before fingerprinting, so a query tagged per request keeps one fingerprint.
+- Name where each exception happened: `role` (`web`, `job`, `runner` or `console`), with the controller action as `route` for requests and `job_class` and `queue` for jobs, including unhandled job failures.
+- Keep the origin of library frames in backtraces (`gems/activerecord-8.1.4/lib/…`, `ruby/4.0.0/…`, `ruby/vendor_ruby/…`) so they can no longer pass for the application's own `lib/` files; an application directory that happens to be called `gems/` stays the application's.
+- Stop fingerprinting session settings and transaction control (`SET`, `SHOW`, `RESET`, `DISCARD`, `BEGIN`, `COMMIT`, `ROLLBACK`, `SAVEPOINT`, `RELEASE`, `PRAGMA` and the like), so a per-query `SET LOCAL statement_timeout` no longer reads as an N+1. They still count toward `query_count`.
+
+## 0.3.0 — 2026-10-10
+
+- Enable bounded automatic endpoint sampling for routine successful requests, driven by expiring hosted policies. Preserve slow, failed, repeated-query requests and all other signal kinds from SDK sampling.
+- Report eligible request selection and queue losses with immutable IDs and independent acknowledgments, including report-only batches at zero sampling rates. Keep old-server event delivery compatible until a v1 handshake.
+- Add explicit local opt-out, effective request metadata, bounded control parsing, fork resets, and sampling diagnostics. Local opt-out does not disable server-side sampling or quotas.
+- Separate known local drops from unconfirmed delivery; failed responses may follow partial server acceptance.
+
 ## 0.2.0 — 2026-10-09
 
 - Correlate requests, serialized ActiveJob executions, and mailer attempts with native trace IDs while preserving existing OpenTelemetry context.

data/README.md

@@ -2,11 +2,11 @@
 
 The MIT-licensed Ruby client for [RailsMind](https://railsmind.com). One bounded
 collector for Rails requests, errors, jobs, mailer attempts, Ahoy events, Flipper
-observations, and selected OpenTelemetry spans. Version 0.2.0 is an early release; see
+observations, and selected OpenTelemetry spans. Version 0.3.0 is an early release; see
 [compatibility and validation limits](docs/compatibility.md).
 
 ```ruby
-gem "rails_mind", "~> 0.2.0"
+gem "rails_mind", "~> 0.3.0"
 ```
 
 Install from RubyGems.org and commit your application's `Gemfile.lock` to pin the
@@ -67,4 +67,4 @@ bundle exec ruby script/package_smoke.rb
 ruby script_benchmark.rb
 ```
 
-Business/request/error events are unsampled, but delivery is best effort. Check `RailsMind.stats` for losses. The SDK does not guarantee durable business-event counts; use a transactional outbox when that is required.
+Automatic endpoint sampling is enabled by default. The hosted policy can reduce routine successful requests while preserving failed, slow, or repeated-query requests and all other signal kinds from sampling. Business/error events remain unsampled; bounded queues and delivery are still best effort. See [sampling and coverage](docs/sdk.md#automatic-endpoint-sampling) and inspect `RailsMind.stats`. Use a transactional outbox when durable business-event delivery is required.

data/docs/compatibility.md

@@ -94,3 +94,19 @@ recipient inbox delivery, direct backend jobs, direct provider HTTP sends, and
 durable workflow completeness are outside this contract. Validate the released
 SDK against your app's queue, callback ordering, and delivery adapter. The hosted
 service exposes bounded workflow metadata and leaves recipient delivery unknown.
+
+## 0.3.0 endpoint sampling verification
+
+The Ruby suite passes locally with Ruby 4.0.7 and the locked Rails 8.1.4
+libraries: 105 tests and 1,311 assertions, with no failures or skips. The three
+Node browser-adapter contract tests also pass. These are isolated SDK/library
+tests; they do not boot a Rails application or use a database. CI separately
+runs the packaged-gem installation smoke check on Ruby 4.0.6.
+
+Coverage includes protected signals, deterministic shared selection, endpoint
+rates, local opt-out, malformed/expired controls, monotonic expiry, queue loss,
+immutable reports and acknowledgments, quota-response retries, old-server
+fallback, idle zero-rate report refresh, streamed response bounds, batch byte
+limits, and real process fork resets. Fake transports verify the wire contract;
+deployed server policy behavior, cross-process traffic forecasts, and a
+customer app's production overhead require hosted/staging validation.

data/docs/sdk.md

@@ -3,13 +3,13 @@
 The gem is `rails_mind`, namespace `RailsMind`. Install the MIT-licensed client
 from RubyGems.org and commit the application lockfile to retain the selected
 version. The hosted RailsMind service is a separate, proprietary application.
-Version 0.2.0 is an early release with the [documented compatibility limits](compatibility.md).
+Version 0.3.0 is an early release with the [documented compatibility limits](compatibility.md).
 
 ## Install
 
 ```ruby
 # Customer Gemfile
-gem "rails_mind", "~> 0.2.0"
+gem "rails_mind", "~> 0.3.0"
 ```
 
 ```sh
@@ -180,9 +180,74 @@ RailsMind::Integrations::OpenTelemetry.attach!
 
 Attachment adds one idempotent processor; it does not replace the global provider, sampler, propagator, instrumentation, or other exporters. Its exporter only sanitizes and enqueues locally; the shared worker performs network I/O. Span context is attached at start so identity survives later export. Known safe attributes are allowlisted. Raw span names, exception span events, HTTP bodies/headers, SQL statements, and bind values are not exported to RailsMind. Existing third-party exporters retain their own behavior and privacy settings. [OpenTelemetry exporters](https://opentelemetry.io/docs/languages/ruby/exporters/).
 
-Request durations/counts and error reports come independently from Rails notifications/error reporting, even when detailed traces are sampled out. Request measurements cover controller actions, not assets/health checks rejected before ActionController or streaming-body completion. Repeated-query evidence contains only normalized-query hashes and counts; it is a suspicion, not proof of an N+1. Hash normalization handles common literals, not every SQL grammar. Use Bullet/Prosopite or an explicit query-bound test in development to verify fixes. No production object scans are installed.
+Request measurements and error reports come independently from Rails notifications/error reporting, even when detailed traces are sampled out. Routine requests separately follow the automatic endpoint sampling policy described below. Request measurements cover controller actions, not assets/health checks rejected before ActionController or streaming-body completion. Repeated-query evidence contains normalized-query hashes and counts, plus the shape of each query that repeated: the normalized SQL with string and numeric literals replaced by `?`, comments removed, and at most 400 characters. Set `config.capture_query_shapes = false` to send hashes only. Session settings and transaction control (`SET`, `SHOW`, `BEGIN`, `COMMIT`, `SAVEPOINT` and the like) count toward `query_count` but are never fingerprinted. It is a suspicion, not proof of an N+1. Normalization handles common literals, not every SQL grammar. Use Bullet/Prosopite or an explicit query-bound test in development to verify fixes. No production object scans are installed.
 
-Full metrics/logs SDK integration is deferred; hosted aggregates use unsampled request events and the SDK does not capture application logs.
+Full metrics/logs SDK integration is deferred. Hosted request aggregates describe retained events; sampling can bias their distributions. The SDK does not capture application logs.
+
+## Automatic endpoint sampling
+
+Version 0.3.0 enables automatic endpoint sampling by default. A compatible hosted
+service supplies environment-wide endpoint rates based on its allowance policy.
+The SDK applies them before queueing, reducing serialization, network, and retained
+event volume. Each process uses the same policy and deterministic event-ID hash;
+there are no independent per-process allowance grants. A server-side tightening
+uses the same selection rule, so the two samplers do not multiply probabilities.
+
+Only completed `request` events with status 200–399, a finite duration from zero
+to below 1,000 ms, and fewer than three repeated queries are eligible. Slow,
+failed, unknown-duration, and repeated-query requests remain unsampled, as do
+business events, errors, jobs, mailer observations, visits, flags, checks, and
+OpenTelemetry spans. This preserves their sampling priority, not guaranteed
+delivery: queue bounds, payload limits, outages, and the server's hard daily quota
+still apply. Existing OpenTelemetry samplers remain authoritative; this feature
+does not promise complete distributed traces.
+
+To turn off SDK request selection explicitly:
+
+```ruby
+RailsMind.configure do |config|
+  config.automatic_sampling = false
+end
+```
+
+The SDK communicates this choice on policy and ingestion requests. The
+environment owner's hosted setting separately controls server-side sampling;
+local opt-out does **not** disable it or bypass the daily quota.
+
+The background worker requests `/api/v1/sampling_policy` with the ingest key and
+refreshes at most once per minute after its first request observation, including
+when the current rate is zero. It also receives controls on successful ingestion
+and quota responses. Missing, malformed, or expired policies fall back to rate 1;
+this may exhaust the server allowance during an outage. Expiry uses wall and
+monotonic clocks, with a maximum accepted lifetime of ten minutes. Before a valid
+v1 handshake, old servers and custom transports receive ordinary event-only
+batches and retain normal flush behavior. Forked workers reset queues, policies,
+counters, and report IDs and fetch their own copy of the shared policy.
+
+Retained request properties include SDK-owned `rails_mind_sampling` metadata:
+`version`, `policy_id`, effective `rate`, and `state` (`active`, `unavailable`,
+`expired`, `disabled`, or `protected`). This is the SDK selection decision, not
+proof of server acceptance or a complete traffic denominator.
+
+At most once per minute per process, the worker freezes an identified report of
+routine eligible observations; an explicit flush may send it sooner. Reports
+contain controller/action names and counts only: `observed`, `retained` (selected
+for enqueue), `sampled_out`, and `queue_dropped` (a subset of retained). They never
+contain event properties or user/account identifiers. Counts are observed after
+validation, redaction, and `before_send`; hooks that suppress events, crashes,
+protected requests, and unknown endpoint names are outside this denominator.
+Retained counts are not accepted counts. The resulting request percentiles and
+rates cannot be treated as an unbiased whole-traffic distribution.
+
+Reports have immutable UUIDs/bodies and separate durable acknowledgments. They
+can travel in `events: []` batches without consuming event quota, and an
+acknowledged report does not establish delivery of its accompanying events.
+Counters cover at most 100 endpoint names per window, reports share the normal
+batch byte limit, and each process retains at most five unacknowledged reports.
+Overflows or invalid clock windows mark coverage `limited`; reports older than
+seven days are discarded. This is bounded in-memory observation, not a durable
+request ledger. Unacknowledged reports after a confirmed handshake can cause
+`RailsMind.flush` to return false at its deadline.
 
 ## Jobs and mailer workflows
 
@@ -250,18 +315,20 @@ Add `<meta name="rails-mind-page" content="Invoices#index">` to supply a stable
 
 ## Bounds, privacy, and outages
 
-Defaults are one lazy worker per process, 1,000 queued/in-flight events, 16 KiB per event, at most 100 events and 256 KiB per batch, and a two-second flush interval. Overflow drops the newest event. Retries retain the same encoded bytes/UUIDs; only HTTP 429, 5xx, and transient transport failures retry, at most three retries with bounded backoff. Other 4xx responses drop the batch. Redirects are not followed. Shutdown gets two seconds by default, then discards outstanding telemetry; forks start a fresh child queue rather than resending the parent's buffer.
+Defaults are one lazy worker per process, 1,000 queued/in-flight events, 16 KiB per event, at most 100 events and 256 KiB per batch, and a two-second flush interval. Overflow drops the newest event. Retries retain the same encoded bytes/UUIDs; only HTTP 429, 5xx, and transient transport failures retry, at most three retries with bounded backoff. Other 4xx responses end delivery attempts. A failed response can follow partial server acceptance, so such events are classified as delivery unconfirmed. Redirects are not followed. Shutdown gets two seconds by default, then discards outstanding telemetry; forks start a fresh child queue rather than resending the parent's buffer.
 
 ```ruby
 RailsMind.stats
-# => { enqueued:, delivered:, dropped:, retried:, queued:, in_flight:,
-#      losses: { "event.overflow" => 3, "error.delivery" => 1 } }
+# => { enqueued:, delivered:, dropped:, delivery_unconfirmed:, retried:, queued:, in_flight:,
+#      losses: { "event.overflow" => 3, "error.delivery_unconfirmed" => 1 },
+#      sampling: { observed:, retained:, sampled_out:, queue_dropped:,
+#                  unreported:, reports_lost:, pending_reports:, limited:, policy_state:, policy_id: } }
 RailsMind.flush(timeout: 2)
 ```
 
-These are per-process counters. The doctor prints them for its process; inspect running app stats through your existing operational tooling. The hosted app does not currently receive loss counters. Request/business/error signals are unsampled, but process crashes, rejected payloads, redaction hooks, exhausted retries, or queue pressure can still undercount. This is best-effort monitoring, not an accounting ledger. A future transactional outbox can provide durable business-event delivery; the current implementation does not claim that guarantee.
+These are per-process counters. `dropped` counts known local discards, such as queue overflow or oversize payloads. `delivery_unconfirmed` counts events whose delivery attempts ended without a success response, including interrupted in-flight batches; some may have been accepted by the server. `delivered` counts submitted events in batches with a successful HTTP response; it does not prove server retention or background processing. The doctor prints stats for its process; inspect running app stats through your existing operational tooling. Only the routine-request report above sends queue-loss observations to the hosted app. All signal kinds can still undercount after crashes, rejected payloads, hooks, or outages. This is best-effort monitoring, not an accounting ledger; use a transactional outbox when durable business-event delivery is required.
 
-Redaction occurs before buffering and again after custom `before_send`. Sensitive property names are filtered, nesting/string/collection sizes are capped, emails and common credential forms are scrubbed, exception messages are omitted by default, and backtraces drop machine-home prefixes. Bodies, cookies, credentials, SQL binds, and job arguments are never selected by integrations. Arbitrary user-provided text may contain sensitive content that generic redaction cannot recognize: use small, explicitly reviewed business properties. Opting into exception messages requires reviewing `capture_exception_message` and your custom scrubbing policy.
+Redaction occurs before buffering and again after custom `before_send`. Sensitive property names are filtered, nesting/string/collection sizes are capped, emails and common credential forms are scrubbed (exception messages included), and backtraces drop machine-home prefixes while keeping a `gems/<name>-<version>/` or `ruby/<version>/` origin for library frames. Error events also name where they happened: `role`, plus the controller action as `route` or the job's `job_class` and `queue`. Bodies, cookies, credentials, SQL binds, and job arguments are never selected by integrations. Arbitrary user-provided text may contain sensitive content that generic redaction cannot recognize: use small, explicitly reviewed business properties. Exception messages can contain whatever your code interpolated into them; review your scrubbing policy, or set `config.capture_exception_message = false` to replace every message with a placeholder before it leaves.
 
 The final local synthetic burst benchmark on Ruby 4.0.6 processed 10,000 calls in 1.7612 seconds (176.12 μs/call), delivering all 10,000 with zero drops/retries at the 1,000-event queue bound. This measures local sanitization/enqueue with a no-op transport, not production request overhead or network delivery. Earlier runs did overflow; thread scheduling and downstream speed affect the result. Deterministic tests separately verify overflow losses and bounded shutdown during a hung transport. Run `ruby script_benchmark.rb` on the deployment hardware before setting a production budget.
 

data/lib/generators/rails_mind/install/templates/initializer.rb.tt

@@ -1,4 +1,7 @@
 RailsMind.configure do |config|
+  # Enabled by default: reduce routine successful request events using hosted policy.
+  # Local opt-out does not disable server sampling or the environment daily quota.
+  # config.automatic_sampling = false
   config.token = ENV["RAILS_MIND_KEY"] # Environment-scoped server credential. Never expose to browsers.
 
   # Endpoint defaults to https://railsmind.com; RAILS_MIND_ENDPOINT overrides it.
@@ -11,8 +14,12 @@ RailsMind.configure do |config|
   # config.analytics = false
   # config.flags = false
 
-  # Exception messages are omitted by default. Review before enabling.
-  # config.capture_exception_message = true
+  # Exception messages are scrubbed (emails, credentials) and sent. Set false to replace
+  # them with a placeholder and keep them in your app.
+  # config.capture_exception_message = false
+  # Repeated queries are sent as a fingerprint plus their shape (the normalized SQL with
+  # literals replaced and comments removed). Set false to send the fingerprints only.
+  # config.capture_query_shapes = false
   # config.before_send = ->(event) { event } # Return nil to drop an event.
 end
 

data/lib/rails_mind/client.rb

@@ -40,7 +40,9 @@ module RailsMind
       false
     end
 
-    def capture_exception(error, handled: false, severity: :error, context: {}, source: nil, **_options)
+    LOCATION_KEYS = %i[role route job_class queue].freeze
+
+    def capture_exception(error, handled: false, severity: :error, context: {}, source: nil, location: nil, **_options)
       return false unless config.errors
       # Rails reports an exception through more than one boundary. A weak key map
       # avoids retaining exceptions; identity dedup applies only to this object.
@@ -48,13 +50,14 @@ module RailsMind
         return false if @reported_errors.key?(error)
         @reported_errors[error] = true
       end
-      message = config.capture_exception_message ? @redactor.scrub(error.message.to_s) : "[Message omitted]"
+      message = config.capture_exception_message ? @redactor.scrub_message(error) : "[Message omitted]"
       frames = Array(error.backtrace).first(40).map { |frame| safe_frame(frame) }
       Context.with(context.slice(*Context::KEYS)) do
+        where = (location || {}).slice(*LOCATION_KEYS).select { |_, value| value.is_a?(String) && !value.empty? }.transform_values { |value| value[0, 200] }
         emit(:error, error.class.name, status: "error", properties: {
           exception_class: error.class.name, message: message, backtrace: frames,
           handled: handled, severity: severity.to_s, source: source
-        })
+        }.merge(where))
       end
     rescue StandardError
       false
@@ -104,12 +107,27 @@ module RailsMind
       time.iso8601(6)
     end
 
+    # An installed gem or Ruby's own library keeps its origin ("gems/rack-3.1.8/lib/…",
+    # "ruby/4.0.0/…") so it can never pass for the application's lib/. A gem
+    # directory is name-version, or name-sha for a Bundler git checkout.
+    GEM_FRAME = %r{\A.*/gems/([^/]+-(?:\d[^/]*|\h{7,40}))/}
+    RUBY_FRAME = %r{\A.*?/lib/ruby/((?:site_ruby/|vendor_ruby/)?\d[^/]*)/}
+    APP_RELATIVE = %r{\A(?:app|lib|config|test|spec)/}
+
     def safe_frame(frame)
       frame = frame.to_s
       frame = frame.delete_prefix("#{Rails.root}/") if defined?(Rails) && Rails.respond_to?(:root) && Rails.root
-      # Remove machine/user home paths, retaining source filename + line + method.
-      frame = frame.sub(%r{\A.*?/(app|lib|config|test|spec)/}, '\\1/')
-      frame = frame.sub(%r{\A/(?:[^/:]+/)*([^/:]+:\d+)}, '\\1')
+      if frame.match?(APP_RELATIVE)
+        # Already the application's own path.
+      elsif frame.match?(GEM_FRAME)
+        frame = frame.sub(GEM_FRAME, 'gems/\\1/')
+      elsif frame.match?(RUBY_FRAME)
+        frame = frame.sub(RUBY_FRAME, 'ruby/\\1/')
+      else
+        # Remove machine/user home paths, retaining source filename + line + method.
+        frame = frame.sub(%r{\A.*?/(app|lib|config|test|spec)/}, '\\1/')
+        frame = frame.sub(%r{\A/(?:[^/:]+/)*([^/:]+:\d+)}, '\\1')
+      end
       @redactor.scrub(frame)
     end
 

data/lib/rails_mind/collector.rb

@@ -7,17 +7,28 @@ module RailsMind
 
     def push(event)
       check_process
-      json = JSON.generate(event)
+      event = event.transform_keys(&:to_s)
       @mutex.synchronize do
-        reason = if @closed then :closed
-        elsif json.bytesize > @config.max_event_bytes || json.bytesize + 13 > @config.max_batch_bytes then :oversize
+        if @closed
+          drop(event["kind"], :closed)
+          return false
+        end
+        selected, observation = @sampling.consider(event)
+        unless selected
+          start_worker if @autostart
+          return false
+        end
+        json = JSON.generate(event)
+        reason = if json.bytesize > @config.max_event_bytes || json.bytesize + 13 > @config.max_batch_bytes then :oversize
         elsif @queue.size + @in_flight >= @config.queue_capacity then :overflow
         end
         if reason
-          drop(event[:kind] || event["kind"], reason)
+          @sampling.queue_dropped(observation)
+          drop(event["kind"], reason)
+          start_worker if @autostart
           return false
         end
-        @queue << [ json.freeze, event[:kind] || event["kind"] ]
+        @queue << [ json.freeze, event["kind"] ]
         @stats[:enqueued] += 1
         start_worker if @autostart
         @condition.broadcast if @queue.size >= @config.batch_size
@@ -27,7 +38,7 @@ module RailsMind
 
     def stats
       check_process
-      @mutex.synchronize { @stats.merge(queued: @queue.size, in_flight: @in_flight, losses: @losses.dup) }
+      @mutex.synchronize { @stats.merge(queued: @queue.size, in_flight: @in_flight, losses: @losses.dup, sampling: @sampling.stats) }
     end
 
     def flush(timeout: @config.shutdown_timeout)
@@ -35,9 +46,9 @@ module RailsMind
       deadline = monotonic + timeout
       @mutex.synchronize do
         start_worker
-        @force_flush = true
+        @force_flush = @force_report = true
         @condition.broadcast
-        while @queue.any? || @in_flight.positive?
+        while @queue.any? || @in_flight.positive? || @active_report || @sampling.pending?
           remaining = deadline - monotonic
           return false if remaining <= 0
           @condition.wait(@mutex, remaining)
@@ -58,9 +69,10 @@ module RailsMind
         @mutex.synchronize do
           @queue.each { |_, kind| drop(kind, :shutdown) }
           @queue.clear
-          @active_batch.to_a.each { |_, kind| drop(kind, :shutdown) }
-          @active_batch = nil
+          @active_batch.to_a.each { |_, kind| unconfirmed(kind) }
+          @active_batch = @active_report = nil
           @in_flight = 0
+          @sampling.discard!
         end
       end
       success
@@ -73,13 +85,15 @@ module RailsMind
       @mutex = Mutex.new
       @condition = ConditionVariable.new
       @queue, @losses = [], Hash.new(0)
-      @stats = { enqueued: 0, delivered: 0, dropped: 0, retried: 0 }
+      @stats = { enqueued: 0, delivered: 0, dropped: 0, delivery_unconfirmed: 0, retried: 0 }
+      @sampling = Sampling.new(@config)
       @in_flight = 0
-      @worker = @active_batch = nil
-      @closed = @stopping = @force_flush = false
+      @worker = @active_batch = @active_report = nil
+      @closed = @stopping = @force_flush = @force_report = false
     end
 
-    # Never resend a parent's inherited queue after Puma/worker forks.
+    # No inherited observations, report IDs, policies or locked mutexes survive
+    # a Puma/worker fork. Each child refreshes the same environment-wide policy.
     def check_process
       reset_process if @pid != Process.pid
     end
@@ -94,46 +108,73 @@ module RailsMind
 
     def run
       loop do
-        batch = @mutex.synchronize do
+        refresh = @mutex.synchronize do
           @condition.wait(@mutex, @config.flush_interval) if @queue.size < @config.batch_size && !@force_flush && !@stopping
           return if @stopping && @queue.empty?
-          take_batch
+          due = @sampling.policy_due? && @transport.respond_to?(:fetch_sampling_policy)
+          @sampling.policy_requested! if due
+          due
         end
-        next if batch.empty?
-        result = deliver(batch)
+        refresh_policy if refresh
+        batch, body = @mutex.synchronize { take_batch }
+        next unless body
+        result = deliver(body, report_id: @active_report)
         @mutex.synchronize do
           if result
             @stats[:delivered] += batch.size
           else
-            batch.each { |_, kind| drop(kind, :delivery) }
+            batch.each { |_, kind| unconfirmed(kind) }
           end
           @in_flight = 0
-          @active_batch = nil
-          @force_flush = false if @queue.empty?
+          @active_batch = @active_report = nil
+          @force_flush = false if @queue.empty? && !@sampling.pending?
           @condition.broadcast
         end
       end
     end
 
     def take_batch
+      @sampling.rotate(force: @force_report, byte_budget: @config.max_batch_bytes - 64)
+      report = @sampling.report(force: @force_report)
+      @force_report = false if @sampling.supported?
       batch = []
-      bytes = 13
+      report_json = report && JSON.generate(report)
+      bytes = 13 + (report_json ? 19 + report_json.bytesize : 0)
       while batch.size < @config.batch_size && @queue.any?
         entry = @queue.first
         break if bytes + entry.first.bytesize + (batch.empty? ? 0 : 1) > @config.max_batch_bytes
         bytes += entry.first.bytesize + (batch.empty? ? 0 : 1)
         batch << @queue.shift
       end
+      if batch.empty? && !report
+        @force_flush = false
+        @condition.broadcast
+        return [ [], nil ]
+      end
       @in_flight = batch.size
       @active_batch = batch
-      batch
+      @active_report = report && report["id"]
+      @sampling.report_sent! if report
+      body = '{"events":[' + batch.map(&:first).join(",") + ']'
+      body += ',"sampling_report":' + report_json if report_json
+      [ batch, (body + '}').freeze ]
     end
 
-    def deliver(batch)
-      body = '{"events":[' + batch.map(&:first).join(",") + ']}'
+    def refresh_policy
+      result = @transport.fetch_sampling_policy
+      @mutex.synchronize { @sampling.apply_policy(result.sampling_policy) }
+    rescue StandardError
+      nil # Missing/expired controls fall back to unsampled collection.
+    end
+
+    def deliver(body, report_id:)
       attempts = 0
       loop do
         result = @transport.send_batch(body)
+        @mutex.synchronize do
+          @sampling.apply_policy(result.sampling_policy) if result.respond_to?(:sampling_policy)
+          @sampling.acknowledge(result.sampling_report_ack) if report_id && result.respond_to?(:sampling_report_ack) && result.sampling_report_ack == report_id
+        end
         return true if result.success
         return false unless result.retryable && attempts < @config.max_retries
         delay = result.retry_after || @config.retry_base * (2**attempts) * (0.75 + rand * 0.5)
@@ -142,7 +183,12 @@ module RailsMind
         sleep(delay) if delay.positive?
       end
     rescue StandardError
-      false # Telemetry errors cannot escape into the customer app.
+      false
+    end
+
+    def unconfirmed(kind)
+      @stats[:delivery_unconfirmed] += 1
+      @losses["#{kind}.delivery_unconfirmed"] += 1
     end
 
     def drop(kind, reason)

data/lib/rails_mind/configuration.rb

@@ -9,7 +9,7 @@ module RailsMind
     attr_accessor :endpoint, :token, :release, :apm, :errors, :analytics, :flags,
       :queue_capacity, :batch_size, :max_batch_bytes, :max_event_bytes, :flush_interval,
       :max_retries, :retry_base, :open_timeout, :read_timeout, :shutdown_timeout,
-      :before_send, :capture_exception_message
+      :before_send, :capture_exception_message, :capture_query_shapes, :automatic_sampling
 
     def initialize(env: ENV, root: nil)
       @endpoint = nonblank(env["RAILS_MIND_ENDPOINT"]) || DEFAULT_ENDPOINT
@@ -26,13 +26,21 @@ module RailsMind
       @open_timeout = 1.0
       @read_timeout = 2.0
       @shutdown_timeout = 2.0
-      @capture_exception_message = false
+      @capture_exception_message = true
+      @capture_query_shapes = true
+      @automatic_sampling = true
     end
 
     def enabled? = !token.to_s.strip.empty? && !endpoint.to_s.strip.empty?
 
     def validate!
       return self unless enabled?
+      # capture_exception_message stays unvalidated as in 0.3.0, so an app that
+      # assigned it from ENV still boots after upgrading.
+      %i[automatic_sampling capture_query_shapes].each do |flag|
+        value = public_send(flag)
+        raise ArgumentError, "#{flag} must be true or false" unless value == true || value == false
+      end
       uri = URI(endpoint)
       loopback = %w[localhost 127.0.0.1 ::1].include?(uri.hostname)
       raise ArgumentError, "endpoint must use HTTPS (HTTP only on loopback)" unless uri.is_a?(URI::HTTPS) || (uri.is_a?(URI::HTTP) && loopback)

data/lib/rails_mind/diagnostic.rb

@@ -21,15 +21,16 @@ module RailsMind
         endpoint_present: !@config.endpoint.to_s.strip.empty?,
         release_present: !@config.release.to_s.strip.empty?,
         enabled: @config.enabled?,
+        automatic_sampling: @config.automatic_sampling,
         plan: [
           "Add one initializer. No database migrations or Gemfile changes.",
-          "Requests and jobs: unsampled measurements via Rails notifications; errors via Rails.error.",
+          "Requests: automatic sampling of routine successes; jobs via Rails notifications and errors via Rails.error stay unsampled.",
           detected["ahoy_matey"] ? "Ahoy detected: keep current store; explicitly opt into mirror mode after review." : "Analytics: optionally add ahoy_matey; choose hosted-only store or existing database store.",
           detected["opentelemetry-sdk"] ? "OpenTelemetry detected: attach RailsMind exporter after existing SDK setup; preserve sampler and exporters." : "APM traces: optionally add selected OpenTelemetry libraries and explicitly configure one provider.",
           detected["flipper"] ? "Flipper detected: observe notifications only; preserve adapter, groups, targeting and cache." : "Flags: Flipper is optional. No flag configuration is created.",
           "Existing error reporters stay installed. RailsMind deduplicates its own collector only.",
           "Browser: optional Ahoy adapter; enable only after checking existing pageview handlers.",
-          "Business/request/error signals are not sampled; bounded queue losses remain possible. Inspect RailsMind.stats."
+          "Business/error signals are not sampled; routine requests follow the hosted policy. Local opt-out does not disable server sampling or quotas. Inspect RailsMind.stats for queue losses and unconfirmed delivery."
         ],
         limitations: [ "Compatibility applies to tested versions only; this is not a historical import.",
           "No network check is performed and credential values are never printed." ]

data/lib/rails_mind/integrations/rails.rb

@@ -1,11 +1,38 @@
 require "active_support/notifications"
+require "active_support/inflector"
 require "digest"
 
 module RailsMind
   module Integrations
+    # Where an exception happened, from the execution context Rails hands its
+    # error subscribers: the controller action, the job, or the kind of process.
+    # A runner script wins over a job it ran inline, whose context Rails leaves set.
+    def self.location(context, source)
+      controller, job = context[:controller], context[:job]
+      if source == "application.runner.railties"
+        { role: "runner" }
+      elsif controller.respond_to?(:action_name)
+        { role: "web", route: "#{controller.class.name}##{controller.action_name}" }
+      elsif job.respond_to?(:queue_name)
+        { role: "job", job_class: job.class.name, queue: job.queue_name.to_s }
+      elsif defined?(::Rails::Console)
+        { role: "console" }
+      else
+        {}
+      end
+    end
+
+    # The routed action of a request, named the way request events name it.
+    def self.request_location(env)
+      params = env["action_dispatch.request.path_parameters"]
+      return { role: "web" } unless params.is_a?(Hash) && params[:controller] && params[:action]
+      { role: "web", route: "#{ActiveSupport::Inflector.camelize(params[:controller].to_s)}Controller##{params[:action]}" }
+    end
+
     class ErrorSubscriber
       def report(error, handled:, severity:, context:, source: nil, **options)
-        RailsMind.capture_exception(error, handled: handled, severity: severity, context: context, source: source)
+        RailsMind.capture_exception(error, handled: handled, severity: severity, context: context, source: source,
+          location: Integrations.location(context, source))
       end
     end
 
@@ -16,12 +43,12 @@ module RailsMind
         Context.with(request_id: env["action_dispatch.request_id"] || SecureRandom.uuid) do
           Context.with_trace do
             previous = Thread.current[:rails_mind_queries]
-            Thread.current[:rails_mind_queries] = { count: 0, fingerprints: Hash.new(0) }
+            Thread.current[:rails_mind_queries] = { count: 0, fingerprints: Hash.new(0), shapes: {} }
             @app.call(env)
           rescue StandardError => error
             # Capture identity before this request boundary unwinds. Object
             # dedup still leaves other error reporters unaffected.
-            RailsMind.capture_exception(error, source: "rack")
+            RailsMind.capture_exception(error, source: "rack", location: Integrations.request_location(env))
             raise
           ensure
             Thread.current[:rails_mind_queries] = previous
@@ -74,6 +101,30 @@ module RailsMind
     end
 
     module RailsNotifications
+      SHAPE_CHARS = 400
+      # The server keeps a shape of at most 1000 bytes; redaction can lengthen one slightly.
+      SHAPE_BYTES = 900
+      # Session settings and transaction control are bookkeeping, not reads: a
+      # per-query `SET LOCAL statement_timeout` must never look like an N+1.
+      BOOKKEEPING = /\A(?:SET|SHOW|RESET|DISCARD|BEGIN|START TRANSACTION|COMMIT|END|ABORT|ROLLBACK|SAVEPOINT|RELEASE|PRAGMA)\b/i
+      # One pass over the statement, so a quote inside a comment or a comment
+      # marker inside a string cannot derail it: strings (doubled or backslash-
+      # escaped quotes, E'' and $$ forms, and one left open where the statement
+      # was cut), numbers and hex literals become ?, and comments go. Double
+      # quotes are identifiers, except on MySQL, where they can be strings.
+      LEXEME = /(?<!\w)[Ee]'(?:[^'\\]|\\.|'')*'?|'(?:[^'\\]|\\.|'')*'?|\$(\w*)\$.*?(?:\$\1\$|\z)|"(?:[^"\\]|\\.|"")*"?|`[^`]*`?|--[^\n]*|\/\*.*?(?:\*\/|\z)|\b0[xX]\h+\b|\b\d+(?:\.\d+)?(?:[eE][-+]?\d+)?\b/m
+
+      def self.normalize(sql, mysql: false)
+        sql.scrub("?").gsub(LEXEME) do |token|
+          case token[0]
+          when "-", "/" then " "
+          when "`" then token
+          when '"' then mysql ? "?" : token
+          else "?"
+          end
+        end.gsub(/\s+/, " ").strip
+      end
+
       class << self
         def install!
           return if @subscriptions
@@ -83,20 +134,33 @@ module RailsMind
             queries = Thread.current[:rails_mind_queries]
             counts = queries&.fetch(:fingerprints, {}) || {}
             status = payload[:status] || (payload[:exception] ? 500 : 200)
+            top = counts.sort_by { |_, count| -count }.first(10).to_h
+            properties = { controller: payload[:controller], action: payload[:action],
+              method: payload[:method], view_runtime: payload[:view_runtime], db_runtime: payload[:db_runtime],
+              query_count: queries&.fetch(:count), repeated_query_count: counts.values.sum { |n| [ n - 1, 0 ].max },
+              query_fingerprints: top }
+            # Shapes go only for the queries that repeated: that is what a finding is about.
+            if RailsMind.client.config.capture_query_shapes
+              shapes = queries&.fetch(:shapes, {}) || {}
+              properties[:query_shapes] = top.filter_map { |fingerprint, count| [ fingerprint, shapes[fingerprint] ] if count >= 2 && shapes[fingerprint] }.to_h
+            end
             RailsMind.client.emit(:request, "#{payload[:controller]}##{payload[:action]}", duration_ms: event.duration,
-              status: status.to_s, properties: { controller: payload[:controller], action: payload[:action],
-                method: payload[:method], view_runtime: payload[:view_runtime], db_runtime: payload[:db_runtime],
-                query_count: queries&.fetch(:count), repeated_query_count: counts.values.sum { |n| [ n - 1, 0 ].max },
-                query_fingerprints: counts.sort_by { |_, count| -count }.first(10).to_h })
+              status: status.to_s, properties: properties)
           end
           subscribe("sql.active_record") do |event|
             queries = Thread.current[:rails_mind_queries]
             next unless queries && !event.payload[:cached] && !%w[SCHEMA TRANSACTION].include?(event.payload[:name])
             queries[:count] += 1
-            # Normalize locally, transmit only one-way hashes; never SQL or binds.
-            normalized = event.payload[:sql].to_s[0, 16_384].gsub(/'(?:[^']|'')*'/, "?").gsub(/\b\d+\b/, "?").gsub(/\s+/, " ")
+            # Comments (query log tags, Marginalia) and literals go before hashing, so one
+            # query has one fingerprint. Binds are never read. The shape is this same
+            # normalized text, capped, and leaves only when capture_query_shapes allows.
+            mysql = event.payload[:connection].respond_to?(:adapter_name) && event.payload[:connection].adapter_name.to_s.match?(/mysql|trilogy/i)
+            normalized = RailsNotifications.normalize(event.payload[:sql].to_s[0, 16_384], mysql: mysql)
+            next if normalized.match?(BOOKKEEPING)
             fingerprint = Digest::SHA256.hexdigest(normalized)[0, 24]
-            queries[:fingerprints][fingerprint] += 1 if queries[:fingerprints].key?(fingerprint) || queries[:fingerprints].size < 100
+            next unless queries[:fingerprints].key?(fingerprint) || queries[:fingerprints].size < 100
+            queries[:fingerprints][fingerprint] += 1
+            (queries[:shapes] ||= {})[fingerprint] ||= normalized[0, SHAPE_CHARS].byteslice(0, SHAPE_BYTES).scrub("") if RailsMind.client.config.capture_query_shapes
           end
           %w[enqueue.active_job enqueue_at.active_job perform_start.active_job perform.active_job enqueue_retry.active_job retry_stopped.active_job discard.active_job].each do |name|
             subscribe(name) do |event|
@@ -128,7 +192,10 @@ module RailsMind
                   kind = enqueue || operation == "perform_start" ? :span : :job
                   RailsMind.client.emit(kind, job.class.name, duration_ms: event.duration, status: status,
                     properties: job_properties(job, operation))
-                  RailsMind.capture_exception(error, source: "active_job") if error
+                  if error
+                    RailsMind.capture_exception(error, source: "active_job",
+                      location: { role: "job", job_class: job.class.name, queue: job.queue_name.to_s })
+                  end
                 end
               end
             end

data/lib/rails_mind/redactor.rb

@@ -19,6 +19,22 @@ module RailsMind
       end
     end
 
+    # Exception messages carry what generic scrubbing cannot recognize: database
+    # errors quote the row, key or input that failed (and the SQL with its
+    # literals), and Ruby prints hashes and keyword arguments with their values.
+    DATABASE_ERROR = /\A(?:PG|Mysql2|Trilogy|SQLite3)::|StatementInvalid|RecordNotUnique|NotNullViolation|ForeignKeyViolation|ValueTooLong|RangeError\z/
+    SENSITIVE_VALUE = /((?:"|:)?(?:password|passwd|secret|token|api[_-]?key|access[_-]?key|private[_-]?key|authorization|credential|cookie|session)\w*"?\s*(?:=>|:)\s*)(?:"(?:[^"\\]|\\.)*"|'(?:[^'\\]|\\.)*'|[^\s,}\])]+)/i
+
+    def scrub_message(error)
+      text = error.message.to_s.encode("UTF-8", invalid: :replace, undef: :replace, replace: "�")
+      text = text.gsub(SENSITIVE_VALUE, '\\1"[FILTERED]"')
+        .gsub(/(Failing row contains )\([^\n]*\)/, "\\1(…)")
+        .gsub(/(Key \([^)\n]*\)=)\([^\n]*\)/, "\\1(…)")
+        .gsub(/(invalid input (?:syntax|value) for [^:\n]*: )"[^"\n]*"/, '\\1"…"')
+      text = text.gsub(/'(?:[^'\\]|\\.|'')*'?/, "?") if database_error?(error)
+      scrub(text)
+    end
+
     def scrub(value)
       value.encode("UTF-8", invalid: :replace, undef: :replace, replace: "�")
         .gsub(/\b[A-Z0-9._%+\-]+@[A-Z0-9.\-]+\.[A-Z]{2,}\b/i, "[EMAIL]")
@@ -28,5 +44,11 @@ module RailsMind
         .gsub(%r{(https?://[^\s?#]+)[?#][^\s]*}, '\\1')
         .slice(0, MAX_STRING)
     end
+
+    private
+
+    def database_error?(error)
+      error.class.ancestors.any? { |ancestor| ancestor.name.to_s.match?(DATABASE_ERROR) }
+    end
   end
 end

data/lib/rails_mind/sampling.rb

@@ -0,0 +1,211 @@
+require "json"
+require "securerandom"
+require "time"
+require "digest"
+
+module RailsMind
+  # Called under Collector's mutex. Decisions and queue-loss accounting share
+  # the same critical section as report rotation; reports are immutable once cut.
+  class Sampling
+    REPORT_INTERVAL = 60
+    POLICY_INTERVAL = 60
+    MAX_POLICY_TTL = 600
+    MAX_ENDPOINTS = 100
+    MAX_PENDING = 5
+    MAX_COUNT = 1_000_000_000
+    MAX_REPORT_AGE = 7 * 86_400
+    ENDPOINT = /\A[A-Z][A-Za-z0-9_]*(?:::[A-Z][A-Za-z0-9_]*)*#[A-Za-z_][A-Za-z0-9_!?]*\z/
+    UUID = /\A[0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12}\z/i
+
+    def initialize(config, clock: -> { Time.now.utc }, monotonic: -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) })
+      @config, @clock, @monotonic = config, clock, monotonic
+      @counts, @reports = {}, []
+      @from = @clock.call
+      @last_rotation = @monotonic.call
+      @next_policy_at = @next_report_at = 0
+      @policy = nil
+      @activated = false
+      @limited = false
+      @policy_ids = []
+      @stats = { observed: 0, retained: 0, sampled_out: 0, queue_dropped: 0, unreported: 0, reports_lost: 0 }
+    end
+
+    def consider(event)
+      return [ true, nil ] unless event["kind"] == "request"
+      @activated = true
+      rate, state, policy_id = effective_policy
+      name = event["name"]
+      unless eligible?(event) && name.is_a?(String) && name.bytesize <= 200 && name.match?(ENDPOINT)
+        annotate(event, 1.0, "protected", policy_id)
+        return [ true, nil ]
+      end
+      if !@counts.key?(name) && @counts.size >= MAX_ENDPOINTS
+        @stats[:unreported] += 1
+        @limited = true
+        annotate(event, 1.0, "unavailable", policy_id)
+        return [ true, nil ]
+      end
+      count = @counts[name] ||= { "name" => name, "observed" => 0, "retained" => 0, "sampled_out" => 0, "queue_dropped" => 0 }
+      if count["observed"] >= MAX_COUNT
+        @stats[:unreported] += 1
+        @limited = true
+        annotate(event, 1.0, "unavailable", policy_id)
+        return [ true, nil ]
+      end
+      rate = @policy[:endpoints].fetch(name, rate) if state == "active"
+      selected = rate >= 1 || (rate.positive? && fraction(event["id"]) < rate)
+      count["observed"] += 1
+      count[selected ? "retained" : "sampled_out"] += 1
+      @stats[:observed] += 1
+      @stats[selected ? :retained : :sampled_out] += 1
+      @policy_ids << (policy_id || state) unless @policy_ids.include?(policy_id || state) || @policy_ids.size >= 2
+      annotate(event, rate, state, policy_id) if selected
+      [ selected, count ]
+    end
+
+    def queue_dropped(count)
+      return unless count
+      count["queue_dropped"] += 1
+      @stats[:queue_dropped] += 1
+    end
+
+    def apply_policy(value)
+      return false unless value.is_a?(Hash) && value["version"] == 1 && boolean?(value["enabled"])
+      id = value["id"]
+      return false unless id.is_a?(String) && id.bytesize.between?(1, 200) && !id.match?(/\p{Cc}/)
+      issued = parse_time(value["issued_at"])
+      expires = parse_time(value["expires_at"])
+      now = @clock.call
+      return false unless issued && expires && issued <= now + 300 && expires > now && expires > issued && expires - issued <= MAX_POLICY_TTL
+      return false if @policy && issued < @policy[:issued_at]
+      rate = value["default_rate"]
+      rows = value["endpoints"]
+      return false unless valid_rate?(rate) && rows.is_a?(Array) && rows.size <= MAX_ENDPOINTS
+      endpoints = {}
+      rows.each do |row|
+        return false unless row.is_a?(Hash) && row["name"].is_a?(String) && row["name"].bytesize.between?(1, 800) && row["name"].length <= 200 && !row["name"].match?(/\p{Cc}/) && valid_rate?(row["rate"])
+        return false if endpoints.key?(row["name"])
+        endpoints[row["name"]] = row["rate"].to_f
+      end
+      @policy = { id: id, issued_at: issued, expires_at: expires, deadline: @monotonic.call + [ expires - now, MAX_POLICY_TTL ].min,
+        enabled: value["enabled"], default_rate: rate.to_f, endpoints: endpoints }
+      true
+    rescue ArgumentError, EncodingError
+      false
+    end
+
+    def policy_due?
+      @activated && @monotonic.call >= @next_policy_at
+    end
+
+    def policy_requested!
+      @next_policy_at = @monotonic.call + POLICY_INTERVAL
+    end
+
+    def rotate(force: false, byte_budget:)
+      return unless supported?
+      now = @clock.call
+      while @reports.first && Time.iso8601(@reports.first.fetch("from")) < now - MAX_REPORT_AGE
+        @reports.shift
+        @stats[:reports_lost] += 1
+        @limited = true
+      end
+      return if @counts.empty? || (!force && @monotonic.call - @last_rotation < REPORT_INTERVAL)
+      if now < @from || now - @from > 86_400
+        @stats[:unreported] += @counts.values.sum { |entry| entry["observed"] }
+        @limited = true
+      else
+        rows = []
+        template = { "version" => 1, "limited" => @limited, "id" => SecureRandom.uuid, "from" => @from.iso8601(6), "to" => now.iso8601(6),
+          "enabled" => @config.automatic_sampling, "policy_id" => @policy_ids.size == 1 ? @policy_ids.first : "mixed", "endpoints" => rows }
+        @counts.each_value do |entry|
+          candidate = template.merge("endpoints" => rows + [ entry ])
+          if JSON.generate(candidate).bytesize > byte_budget
+            store_report(template.merge("endpoints" => rows)) if rows.any?
+            rows = []
+            template = template.merge("id" => SecureRandom.uuid)
+          end
+          if JSON.generate(template.merge("endpoints" => [ entry ])).bytesize <= byte_budget
+            rows << entry.dup
+          else
+            @stats[:unreported] += entry["observed"]
+            @limited = true
+          end
+        end
+        store_report(template.merge("endpoints" => rows)) if rows.any?
+      end
+      @counts, @policy_ids = {}, []
+      @from, @last_rotation = now, @monotonic.call
+    end
+
+    def report(force: false)
+      @reports.first if force || @monotonic.call >= @next_report_at
+    end
+
+    def report_sent!
+      @next_report_at = @monotonic.call + REPORT_INTERVAL
+    end
+
+    def acknowledge(id)
+      return unless id.is_a?(String) && id.match?(UUID)
+      removed = @reports.reject! { |report| report["id"] == id }
+      @next_report_at = 0 if removed && @reports.any?
+    end
+
+    def discard!
+      @stats[:reports_lost] += @reports.size
+      @stats[:unreported] += @counts.values.sum { |entry| entry["observed"] }
+      @reports, @counts = [], {}
+    end
+
+    def supported? = !@policy.nil?
+    def pending? = supported? && (@reports.any? || @counts.any?)
+    def stats = @stats.merge(limited: @limited, pending_reports: @reports.size, policy_state: effective_policy[1], policy_id: @policy&.fetch(:id))
+
+    private
+
+    def eligible?(event)
+      status, duration = event["status"], event["duration_ms"]
+      repeated = event.fetch("properties", {})["repeated_query_count"]
+      status.is_a?(String) && status.match?(/\A[23][0-9]{2}\z/) && duration.is_a?(Numeric) && duration.finite? && duration >= 0 && duration < 1000 && !(repeated.is_a?(Numeric) && repeated >= 3)
+    end
+
+    def effective_policy
+      return [ 1.0, "disabled", @policy&.fetch(:id) ] unless @config.automatic_sampling
+      return [ 1.0, "unavailable", nil ] unless @policy
+      return [ 1.0, "expired", @policy[:id] ] if @monotonic.call >= @policy[:deadline] || @clock.call >= @policy[:expires_at]
+      return [ 1.0, "disabled", @policy[:id] ] unless @policy[:enabled]
+      [ @policy[:default_rate], "active", @policy[:id] ]
+    end
+
+    def annotate(event, rate, state, policy_id)
+      event["properties"] = event.fetch("properties", {}).merge("rails_mind_sampling" => {
+        "version" => 1, "policy_id" => policy_id, "rate" => rate, "state" => state })
+    end
+
+    def store_report(report)
+      if @reports.size >= MAX_PENDING
+        @reports.shift
+        @stats[:reports_lost] += 1
+        @limited = true
+      end
+      report["limited"] ||= @limited
+      @reports << deep_freeze(JSON.parse(JSON.generate(report)))
+    end
+
+    def deep_freeze(value)
+      case value
+      when Hash then value.each { |key, child| deep_freeze(key); deep_freeze(child) }
+      when Array then value.each { |child| deep_freeze(child) }
+      end
+      value.freeze
+    end
+
+    def fraction(id) = Digest::SHA256.digest("rails_mind:sampling:v1:#{id.to_s.downcase}").unpack1("Q>") / (2.0**64)
+    def boolean?(value) = value == true || value == false
+    def valid_rate?(value) = (value.is_a?(Float) || value.is_a?(Integer)) && value.finite? && value.between?(0, 1)
+    def parse_time(value)
+      Time.iso8601(value) if value.is_a?(String) && value.bytesize <= 40
+    end
+  end
+end

data/lib/rails_mind/transport.rb

@@ -3,7 +3,8 @@ require "json"
 
 module RailsMind
   class Transport
-    Result = Struct.new(:success, :retryable, :retry_after, keyword_init: true)
+    MAX_RESPONSE_BYTES = 64 * 1024
+    Result = Struct.new(:success, :retryable, :retry_after, :sampling_policy, :sampling_report_ack, keyword_init: true)
     TRANSIENT_ERRORS = [ Timeout::Error, IOError, EOFError, SocketError, Errno::ECONNRESET,
       Errno::ECONNREFUSED, Errno::ETIMEDOUT, Errno::EPIPE, OpenSSL::SSL::SSLError ].freeze
 
@@ -15,28 +16,65 @@ module RailsMind
     end
 
     def send_batch(json)
+      request = Net::HTTP::Post.new(@uri)
+      request.body = json
+      perform(request)
+    end
+
+    def fetch_sampling_policy
+      uri = @uri.dup
+      uri.path = uri.path.sub(%r{/events$}, "/sampling_policy")
+      perform(Net::HTTP::Get.new(uri))
+    end
+
+    private
+
+    def perform(request)
       Context.suppress do
-        request = Net::HTTP::Post.new(@uri)
         request["Authorization"] = "Bearer #{@config.token}"
         request["Content-Type"] = "application/json"
         request["User-Agent"] = "rails_mind/#{VERSION}"
-        request.body = json
+        request["X-RailsMind-Automatic-Sampling"] = @config.automatic_sampling ? "enabled" : "disabled"
         http = @http_factory.call(@uri)
         http.use_ssl = @uri.scheme == "https"
         http.open_timeout = @config.open_timeout
         http.read_timeout = @config.read_timeout
         http.write_timeout = @config.read_timeout
         http.max_retries = 0
-        response = http.start { |connection| connection.request(request) }
+        body, streamed, oversized = +"", false, false
+        response = http.start do |connection|
+          connection.request(request) do |reply|
+            streamed = true
+            reply.read_body do |chunk|
+              if body.bytesize + chunk.bytesize > MAX_RESPONSE_BYTES
+                oversized = true
+              elsif !oversized
+                body << chunk
+              end
+            end
+          end
+        end
+        body = response.body.to_s if !streamed && response.respond_to?(:body)
         code = response.code.to_i
+        control = if !oversized && body.bytesize <= MAX_RESPONSE_BYTES && ((200..299).cover?(code) || code == 429)
+          parse_control(body)
+        else
+          {}
+        end
         Result.new(success: (200..299).cover?(code), retryable: code == 429 || code >= 500,
-          retry_after: retry_after(response["Retry-After"]))
+          retry_after: retry_after(response["Retry-After"]), sampling_policy: control["sampling_policy"],
+          sampling_report_ack: control["sampling_report_ack"])
       end
     rescue *TRANSIENT_ERRORS
       Result.new(success: false, retryable: true)
     end
 
-    private
+    def parse_control(body)
+      value = JSON.parse(body)
+      value.is_a?(Hash) ? value : {}
+    rescue JSON::ParserError, EncodingError
+      {}
+    end
 
     def retry_after(value)
       return nil unless value

data/lib/rails_mind/version.rb

@@ -1,3 +1,3 @@
 module RailsMind
-  VERSION = "0.2.0"
+  VERSION = "0.3.1"
 end

data/lib/rails_mind.rb

@@ -3,6 +3,7 @@ require_relative "rails_mind/configuration"
 require_relative "rails_mind/context"
 require_relative "rails_mind/redactor"
 require_relative "rails_mind/transport"
+require_relative "rails_mind/sampling"
 require_relative "rails_mind/collector"
 require_relative "rails_mind/client"
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rails_mind
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.1
 platform: ruby
 authors:
 - RailsMind
@@ -81,6 +81,7 @@ files:
 - lib/rails_mind/integrations/rails.rb
 - lib/rails_mind/railtie.rb
 - lib/rails_mind/redactor.rb
+- lib/rails_mind/sampling.rb
 - lib/rails_mind/transport.rb
 - lib/rails_mind/version.rb
 - lib/tasks/rails_mind.rake