rails_mind 0.3.0 → 0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 819a570e3f4f65e0fb8bae7de68b1a81ee744bccb77349858c640fcc912e95b4
-  data.tar.gz: 62df4fca70603e28458d277dcd8d734a5de04af7c514f997e15686122a02dbd2
+  metadata.gz: 902c75e77dcb5d76b75de68e62e73fe7a92f38ec05f621084bc6a61c047cb186
+  data.tar.gz: b499762228b099f7d940a3b4aa58ba1d1c73d4a9c53958a02e379982fa08b28b
 SHA512:
-  metadata.gz: 3583084684b6ac4b165864c14fbfe3922fc2f678c8d7aa9383418d95fdc528083bbe2327e5428d167d80e54a28445309eb6243bd3c6c3dac36fc8a67fa9e0630
-  data.tar.gz: e968233e4025ce470a83207214ef886032bff662faea00f824f3aceefad55319aa6d3235b62cc16368c5b8ccec7db86a6914ce1d5cb1a5db2c9bb282a543f0f2
+  metadata.gz: ae733243113a0206980e2b34f2545f5648c00b4e032d32ae2727bd8b5da85d67067970cc73ff1654eef1eb93782457d71cb65c45e9a6bfcde5f6f7fc03985310
+  data.tar.gz: 48d196c72e82519469bc6afba1e47a585012a9ff3f0812c243ddd9b00378f6512f5b7d4f8d62fbd5158a029e2166d35e8d3373588fc3458c87c324848d477ed2

data/CHANGELOG.md

@@ -1,5 +1,14 @@
 # Changelog
 
+## 0.3.1 — unreleased
+
+- Send exception messages by default, scrubbed like every other string (emails, credentials, URL queries), and further for what generic scrubbing misses: database errors lose the failing row, key values, rejected input and quoted SQL literals, and printed hashes or keyword arguments lose the values of sensitive keys. `config.capture_exception_message = false` restores the placeholder. The flag is still not validated, so an app that assigned it from ENV boots as before.
+- Send the shape of each repeated query next to its fingerprint: the normalized SQL with literals replaced by `?`, comments removed, and at most 400 characters and 900 bytes, only for fingerprints that repeated within the request. One pass handles doubled and backslash-escaped quotes, `E''` and `$$` strings, hex numbers, comments containing quotes, a literal left open where a long statement is cut, and MySQL's double-quoted strings. `config.capture_query_shapes = false` sends fingerprints only.
+- Strip SQL comments (query log tags, Marginalia) before fingerprinting, so a query tagged per request keeps one fingerprint.
+- Name where each exception happened: `role` (`web`, `job`, `runner` or `console`), with the controller action as `route` for requests and `job_class` and `queue` for jobs, including unhandled job failures.
+- Keep the origin of library frames in backtraces (`gems/activerecord-8.1.4/lib/…`, `ruby/4.0.0/…`, `ruby/vendor_ruby/…`) so they can no longer pass for the application's own `lib/` files; an application directory that happens to be called `gems/` stays the application's.
+- Stop fingerprinting session settings and transaction control (`SET`, `SHOW`, `RESET`, `DISCARD`, `BEGIN`, `COMMIT`, `ROLLBACK`, `SAVEPOINT`, `RELEASE`, `PRAGMA` and the like), so a per-query `SET LOCAL statement_timeout` no longer reads as an N+1. They still count toward `query_count`.
+
 ## 0.3.0 — 2026-10-10
 
 - Enable bounded automatic endpoint sampling for routine successful requests, driven by expiring hosted policies. Preserve slow, failed, repeated-query requests and all other signal kinds from SDK sampling.

data/docs/sdk.md

@@ -180,7 +180,7 @@ RailsMind::Integrations::OpenTelemetry.attach!
 
 Attachment adds one idempotent processor; it does not replace the global provider, sampler, propagator, instrumentation, or other exporters. Its exporter only sanitizes and enqueues locally; the shared worker performs network I/O. Span context is attached at start so identity survives later export. Known safe attributes are allowlisted. Raw span names, exception span events, HTTP bodies/headers, SQL statements, and bind values are not exported to RailsMind. Existing third-party exporters retain their own behavior and privacy settings. [OpenTelemetry exporters](https://opentelemetry.io/docs/languages/ruby/exporters/).
 
-Request measurements and error reports come independently from Rails notifications/error reporting, even when detailed traces are sampled out. Routine requests separately follow the automatic endpoint sampling policy described below. Request measurements cover controller actions, not assets/health checks rejected before ActionController or streaming-body completion. Repeated-query evidence contains only normalized-query hashes and counts; it is a suspicion, not proof of an N+1. Hash normalization handles common literals, not every SQL grammar. Use Bullet/Prosopite or an explicit query-bound test in development to verify fixes. No production object scans are installed.
+Request measurements and error reports come independently from Rails notifications/error reporting, even when detailed traces are sampled out. Routine requests separately follow the automatic endpoint sampling policy described below. Request measurements cover controller actions, not assets/health checks rejected before ActionController or streaming-body completion. Repeated-query evidence contains normalized-query hashes and counts, plus the shape of each query that repeated: the normalized SQL with string and numeric literals replaced by `?`, comments removed, and at most 400 characters. Set `config.capture_query_shapes = false` to send hashes only. Session settings and transaction control (`SET`, `SHOW`, `BEGIN`, `COMMIT`, `SAVEPOINT` and the like) count toward `query_count` but are never fingerprinted. It is a suspicion, not proof of an N+1. Normalization handles common literals, not every SQL grammar. Use Bullet/Prosopite or an explicit query-bound test in development to verify fixes. No production object scans are installed.
 
 Full metrics/logs SDK integration is deferred. Hosted request aggregates describe retained events; sampling can bias their distributions. The SDK does not capture application logs.
 
@@ -328,7 +328,7 @@ RailsMind.flush(timeout: 2)
 
 These are per-process counters. `dropped` counts known local discards, such as queue overflow or oversize payloads. `delivery_unconfirmed` counts events whose delivery attempts ended without a success response, including interrupted in-flight batches; some may have been accepted by the server. `delivered` counts submitted events in batches with a successful HTTP response; it does not prove server retention or background processing. The doctor prints stats for its process; inspect running app stats through your existing operational tooling. Only the routine-request report above sends queue-loss observations to the hosted app. All signal kinds can still undercount after crashes, rejected payloads, hooks, or outages. This is best-effort monitoring, not an accounting ledger; use a transactional outbox when durable business-event delivery is required.
 
-Redaction occurs before buffering and again after custom `before_send`. Sensitive property names are filtered, nesting/string/collection sizes are capped, emails and common credential forms are scrubbed, exception messages are omitted by default, and backtraces drop machine-home prefixes. Bodies, cookies, credentials, SQL binds, and job arguments are never selected by integrations. Arbitrary user-provided text may contain sensitive content that generic redaction cannot recognize: use small, explicitly reviewed business properties. Opting into exception messages requires reviewing `capture_exception_message` and your custom scrubbing policy.
+Redaction occurs before buffering and again after custom `before_send`. Sensitive property names are filtered, nesting/string/collection sizes are capped, emails and common credential forms are scrubbed (exception messages included), and backtraces drop machine-home prefixes while keeping a `gems/<name>-<version>/` or `ruby/<version>/` origin for library frames. Error events also name where they happened: `role`, plus the controller action as `route` or the job's `job_class` and `queue`. Bodies, cookies, credentials, SQL binds, and job arguments are never selected by integrations. Arbitrary user-provided text may contain sensitive content that generic redaction cannot recognize: use small, explicitly reviewed business properties. Exception messages can contain whatever your code interpolated into them; review your scrubbing policy, or set `config.capture_exception_message = false` to replace every message with a placeholder before it leaves.
 
 The final local synthetic burst benchmark on Ruby 4.0.6 processed 10,000 calls in 1.7612 seconds (176.12 μs/call), delivering all 10,000 with zero drops/retries at the 1,000-event queue bound. This measures local sanitization/enqueue with a no-op transport, not production request overhead or network delivery. Earlier runs did overflow; thread scheduling and downstream speed affect the result. Deterministic tests separately verify overflow losses and bounded shutdown during a hung transport. Run `ruby script_benchmark.rb` on the deployment hardware before setting a production budget.
 

data/lib/generators/rails_mind/install/templates/initializer.rb.tt

@@ -14,8 +14,12 @@ RailsMind.configure do |config|
   # config.analytics = false
   # config.flags = false
 
-  # Exception messages are omitted by default. Review before enabling.
-  # config.capture_exception_message = true
+  # Exception messages are scrubbed (emails, credentials) and sent. Set false to replace
+  # them with a placeholder and keep them in your app.
+  # config.capture_exception_message = false
+  # Repeated queries are sent as a fingerprint plus their shape (the normalized SQL with
+  # literals replaced and comments removed). Set false to send the fingerprints only.
+  # config.capture_query_shapes = false
   # config.before_send = ->(event) { event } # Return nil to drop an event.
 end
 

data/lib/rails_mind/client.rb

@@ -40,7 +40,9 @@ module RailsMind
       false
     end
 
-    def capture_exception(error, handled: false, severity: :error, context: {}, source: nil, **_options)
+    LOCATION_KEYS = %i[role route job_class queue].freeze
+
+    def capture_exception(error, handled: false, severity: :error, context: {}, source: nil, location: nil, **_options)
       return false unless config.errors
       # Rails reports an exception through more than one boundary. A weak key map
       # avoids retaining exceptions; identity dedup applies only to this object.
@@ -48,13 +50,14 @@ module RailsMind
         return false if @reported_errors.key?(error)
         @reported_errors[error] = true
       end
-      message = config.capture_exception_message ? @redactor.scrub(error.message.to_s) : "[Message omitted]"
+      message = config.capture_exception_message ? @redactor.scrub_message(error) : "[Message omitted]"
       frames = Array(error.backtrace).first(40).map { |frame| safe_frame(frame) }
       Context.with(context.slice(*Context::KEYS)) do
+        where = (location || {}).slice(*LOCATION_KEYS).select { |_, value| value.is_a?(String) && !value.empty? }.transform_values { |value| value[0, 200] }
         emit(:error, error.class.name, status: "error", properties: {
           exception_class: error.class.name, message: message, backtrace: frames,
           handled: handled, severity: severity.to_s, source: source
-        })
+        }.merge(where))
       end
     rescue StandardError
       false
@@ -104,12 +107,27 @@ module RailsMind
       time.iso8601(6)
     end
 
+    # An installed gem or Ruby's own library keeps its origin ("gems/rack-3.1.8/lib/…",
+    # "ruby/4.0.0/…") so it can never pass for the application's lib/. A gem
+    # directory is name-version, or name-sha for a Bundler git checkout.
+    GEM_FRAME = %r{\A.*/gems/([^/]+-(?:\d[^/]*|\h{7,40}))/}
+    RUBY_FRAME = %r{\A.*?/lib/ruby/((?:site_ruby/|vendor_ruby/)?\d[^/]*)/}
+    APP_RELATIVE = %r{\A(?:app|lib|config|test|spec)/}
+
     def safe_frame(frame)
       frame = frame.to_s
       frame = frame.delete_prefix("#{Rails.root}/") if defined?(Rails) && Rails.respond_to?(:root) && Rails.root
-      # Remove machine/user home paths, retaining source filename + line + method.
-      frame = frame.sub(%r{\A.*?/(app|lib|config|test|spec)/}, '\\1/')
-      frame = frame.sub(%r{\A/(?:[^/:]+/)*([^/:]+:\d+)}, '\\1')
+      if frame.match?(APP_RELATIVE)
+        # Already the application's own path.
+      elsif frame.match?(GEM_FRAME)
+        frame = frame.sub(GEM_FRAME, 'gems/\\1/')
+      elsif frame.match?(RUBY_FRAME)
+        frame = frame.sub(RUBY_FRAME, 'ruby/\\1/')
+      else
+        # Remove machine/user home paths, retaining source filename + line + method.
+        frame = frame.sub(%r{\A.*?/(app|lib|config|test|spec)/}, '\\1/')
+        frame = frame.sub(%r{\A/(?:[^/:]+/)*([^/:]+:\d+)}, '\\1')
+      end
       @redactor.scrub(frame)
     end
 

data/lib/rails_mind/configuration.rb

@@ -9,7 +9,7 @@ module RailsMind
     attr_accessor :endpoint, :token, :release, :apm, :errors, :analytics, :flags,
       :queue_capacity, :batch_size, :max_batch_bytes, :max_event_bytes, :flush_interval,
       :max_retries, :retry_base, :open_timeout, :read_timeout, :shutdown_timeout,
-      :before_send, :capture_exception_message, :automatic_sampling
+      :before_send, :capture_exception_message, :capture_query_shapes, :automatic_sampling
 
     def initialize(env: ENV, root: nil)
       @endpoint = nonblank(env["RAILS_MIND_ENDPOINT"]) || DEFAULT_ENDPOINT
@@ -26,7 +26,8 @@ module RailsMind
       @open_timeout = 1.0
       @read_timeout = 2.0
       @shutdown_timeout = 2.0
-      @capture_exception_message = false
+      @capture_exception_message = true
+      @capture_query_shapes = true
       @automatic_sampling = true
     end
 
@@ -34,7 +35,12 @@ module RailsMind
 
     def validate!
       return self unless enabled?
-      raise ArgumentError, "automatic_sampling must be true or false" unless automatic_sampling == true || automatic_sampling == false
+      # capture_exception_message stays unvalidated as in 0.3.0, so an app that
+      # assigned it from ENV still boots after upgrading.
+      %i[automatic_sampling capture_query_shapes].each do |flag|
+        value = public_send(flag)
+        raise ArgumentError, "#{flag} must be true or false" unless value == true || value == false
+      end
       uri = URI(endpoint)
       loopback = %w[localhost 127.0.0.1 ::1].include?(uri.hostname)
       raise ArgumentError, "endpoint must use HTTPS (HTTP only on loopback)" unless uri.is_a?(URI::HTTPS) || (uri.is_a?(URI::HTTP) && loopback)

data/lib/rails_mind/integrations/rails.rb

@@ -1,11 +1,38 @@
 require "active_support/notifications"
+require "active_support/inflector"
 require "digest"
 
 module RailsMind
   module Integrations
+    # Where an exception happened, from the execution context Rails hands its
+    # error subscribers: the controller action, the job, or the kind of process.
+    # A runner script wins over a job it ran inline, whose context Rails leaves set.
+    def self.location(context, source)
+      controller, job = context[:controller], context[:job]
+      if source == "application.runner.railties"
+        { role: "runner" }
+      elsif controller.respond_to?(:action_name)
+        { role: "web", route: "#{controller.class.name}##{controller.action_name}" }
+      elsif job.respond_to?(:queue_name)
+        { role: "job", job_class: job.class.name, queue: job.queue_name.to_s }
+      elsif defined?(::Rails::Console)
+        { role: "console" }
+      else
+        {}
+      end
+    end
+
+    # The routed action of a request, named the way request events name it.
+    def self.request_location(env)
+      params = env["action_dispatch.request.path_parameters"]
+      return { role: "web" } unless params.is_a?(Hash) && params[:controller] && params[:action]
+      { role: "web", route: "#{ActiveSupport::Inflector.camelize(params[:controller].to_s)}Controller##{params[:action]}" }
+    end
+
     class ErrorSubscriber
       def report(error, handled:, severity:, context:, source: nil, **options)
-        RailsMind.capture_exception(error, handled: handled, severity: severity, context: context, source: source)
+        RailsMind.capture_exception(error, handled: handled, severity: severity, context: context, source: source,
+          location: Integrations.location(context, source))
       end
     end
 
@@ -16,12 +43,12 @@ module RailsMind
         Context.with(request_id: env["action_dispatch.request_id"] || SecureRandom.uuid) do
           Context.with_trace do
             previous = Thread.current[:rails_mind_queries]
-            Thread.current[:rails_mind_queries] = { count: 0, fingerprints: Hash.new(0) }
+            Thread.current[:rails_mind_queries] = { count: 0, fingerprints: Hash.new(0), shapes: {} }
             @app.call(env)
           rescue StandardError => error
             # Capture identity before this request boundary unwinds. Object
             # dedup still leaves other error reporters unaffected.
-            RailsMind.capture_exception(error, source: "rack")
+            RailsMind.capture_exception(error, source: "rack", location: Integrations.request_location(env))
             raise
           ensure
             Thread.current[:rails_mind_queries] = previous
@@ -74,6 +101,30 @@ module RailsMind
     end
 
     module RailsNotifications
+      SHAPE_CHARS = 400
+      # The server keeps a shape of at most 1000 bytes; redaction can lengthen one slightly.
+      SHAPE_BYTES = 900
+      # Session settings and transaction control are bookkeeping, not reads: a
+      # per-query `SET LOCAL statement_timeout` must never look like an N+1.
+      BOOKKEEPING = /\A(?:SET|SHOW|RESET|DISCARD|BEGIN|START TRANSACTION|COMMIT|END|ABORT|ROLLBACK|SAVEPOINT|RELEASE|PRAGMA)\b/i
+      # One pass over the statement, so a quote inside a comment or a comment
+      # marker inside a string cannot derail it: strings (doubled or backslash-
+      # escaped quotes, E'' and $$ forms, and one left open where the statement
+      # was cut), numbers and hex literals become ?, and comments go. Double
+      # quotes are identifiers, except on MySQL, where they can be strings.
+      LEXEME = /(?<!\w)[Ee]'(?:[^'\\]|\\.|'')*'?|'(?:[^'\\]|\\.|'')*'?|\$(\w*)\$.*?(?:\$\1\$|\z)|"(?:[^"\\]|\\.|"")*"?|`[^`]*`?|--[^\n]*|\/\*.*?(?:\*\/|\z)|\b0[xX]\h+\b|\b\d+(?:\.\d+)?(?:[eE][-+]?\d+)?\b/m
+
+      def self.normalize(sql, mysql: false)
+        sql.scrub("?").gsub(LEXEME) do |token|
+          case token[0]
+          when "-", "/" then " "
+          when "`" then token
+          when '"' then mysql ? "?" : token
+          else "?"
+          end
+        end.gsub(/\s+/, " ").strip
+      end
+
       class << self
         def install!
           return if @subscriptions
@@ -83,20 +134,33 @@ module RailsMind
             queries = Thread.current[:rails_mind_queries]
             counts = queries&.fetch(:fingerprints, {}) || {}
             status = payload[:status] || (payload[:exception] ? 500 : 200)
+            top = counts.sort_by { |_, count| -count }.first(10).to_h
+            properties = { controller: payload[:controller], action: payload[:action],
+              method: payload[:method], view_runtime: payload[:view_runtime], db_runtime: payload[:db_runtime],
+              query_count: queries&.fetch(:count), repeated_query_count: counts.values.sum { |n| [ n - 1, 0 ].max },
+              query_fingerprints: top }
+            # Shapes go only for the queries that repeated: that is what a finding is about.
+            if RailsMind.client.config.capture_query_shapes
+              shapes = queries&.fetch(:shapes, {}) || {}
+              properties[:query_shapes] = top.filter_map { |fingerprint, count| [ fingerprint, shapes[fingerprint] ] if count >= 2 && shapes[fingerprint] }.to_h
+            end
             RailsMind.client.emit(:request, "#{payload[:controller]}##{payload[:action]}", duration_ms: event.duration,
-              status: status.to_s, properties: { controller: payload[:controller], action: payload[:action],
-                method: payload[:method], view_runtime: payload[:view_runtime], db_runtime: payload[:db_runtime],
-                query_count: queries&.fetch(:count), repeated_query_count: counts.values.sum { |n| [ n - 1, 0 ].max },
-                query_fingerprints: counts.sort_by { |_, count| -count }.first(10).to_h })
+              status: status.to_s, properties: properties)
           end
           subscribe("sql.active_record") do |event|
             queries = Thread.current[:rails_mind_queries]
             next unless queries && !event.payload[:cached] && !%w[SCHEMA TRANSACTION].include?(event.payload[:name])
             queries[:count] += 1
-            # Normalize locally, transmit only one-way hashes; never SQL or binds.
-            normalized = event.payload[:sql].to_s[0, 16_384].gsub(/'(?:[^']|'')*'/, "?").gsub(/\b\d+\b/, "?").gsub(/\s+/, " ")
+            # Comments (query log tags, Marginalia) and literals go before hashing, so one
+            # query has one fingerprint. Binds are never read. The shape is this same
+            # normalized text, capped, and leaves only when capture_query_shapes allows.
+            mysql = event.payload[:connection].respond_to?(:adapter_name) && event.payload[:connection].adapter_name.to_s.match?(/mysql|trilogy/i)
+            normalized = RailsNotifications.normalize(event.payload[:sql].to_s[0, 16_384], mysql: mysql)
+            next if normalized.match?(BOOKKEEPING)
             fingerprint = Digest::SHA256.hexdigest(normalized)[0, 24]
-            queries[:fingerprints][fingerprint] += 1 if queries[:fingerprints].key?(fingerprint) || queries[:fingerprints].size < 100
+            next unless queries[:fingerprints].key?(fingerprint) || queries[:fingerprints].size < 100
+            queries[:fingerprints][fingerprint] += 1
+            (queries[:shapes] ||= {})[fingerprint] ||= normalized[0, SHAPE_CHARS].byteslice(0, SHAPE_BYTES).scrub("") if RailsMind.client.config.capture_query_shapes
           end
           %w[enqueue.active_job enqueue_at.active_job perform_start.active_job perform.active_job enqueue_retry.active_job retry_stopped.active_job discard.active_job].each do |name|
             subscribe(name) do |event|
@@ -128,7 +192,10 @@ module RailsMind
                   kind = enqueue || operation == "perform_start" ? :span : :job
                   RailsMind.client.emit(kind, job.class.name, duration_ms: event.duration, status: status,
                     properties: job_properties(job, operation))
-                  RailsMind.capture_exception(error, source: "active_job") if error
+                  if error
+                    RailsMind.capture_exception(error, source: "active_job",
+                      location: { role: "job", job_class: job.class.name, queue: job.queue_name.to_s })
+                  end
                 end
               end
             end

data/lib/rails_mind/redactor.rb

@@ -19,6 +19,22 @@ module RailsMind
       end
     end
 
+    # Exception messages carry what generic scrubbing cannot recognize: database
+    # errors quote the row, key or input that failed (and the SQL with its
+    # literals), and Ruby prints hashes and keyword arguments with their values.
+    DATABASE_ERROR = /\A(?:PG|Mysql2|Trilogy|SQLite3)::|StatementInvalid|RecordNotUnique|NotNullViolation|ForeignKeyViolation|ValueTooLong|RangeError\z/
+    SENSITIVE_VALUE = /((?:"|:)?(?:password|passwd|secret|token|api[_-]?key|access[_-]?key|private[_-]?key|authorization|credential|cookie|session)\w*"?\s*(?:=>|:)\s*)(?:"(?:[^"\\]|\\.)*"|'(?:[^'\\]|\\.)*'|[^\s,}\])]+)/i
+
+    def scrub_message(error)
+      text = error.message.to_s.encode("UTF-8", invalid: :replace, undef: :replace, replace: "�")
+      text = text.gsub(SENSITIVE_VALUE, '\\1"[FILTERED]"')
+        .gsub(/(Failing row contains )\([^\n]*\)/, "\\1(…)")
+        .gsub(/(Key \([^)\n]*\)=)\([^\n]*\)/, "\\1(…)")
+        .gsub(/(invalid input (?:syntax|value) for [^:\n]*: )"[^"\n]*"/, '\\1"…"')
+      text = text.gsub(/'(?:[^'\\]|\\.|'')*'?/, "?") if database_error?(error)
+      scrub(text)
+    end
+
     def scrub(value)
       value.encode("UTF-8", invalid: :replace, undef: :replace, replace: "�")
         .gsub(/\b[A-Z0-9._%+\-]+@[A-Z0-9.\-]+\.[A-Z]{2,}\b/i, "[EMAIL]")
@@ -28,5 +44,11 @@ module RailsMind
         .gsub(%r{(https?://[^\s?#]+)[?#][^\s]*}, '\\1')
         .slice(0, MAX_STRING)
     end
+
+    private
+
+    def database_error?(error)
+      error.class.ancestors.any? { |ancestor| ancestor.name.to_s.match?(DATABASE_ERROR) }
+    end
   end
 end

data/lib/rails_mind/version.rb

@@ -1,3 +1,3 @@
 module RailsMind
-  VERSION = "0.3.0"
+  VERSION = "0.3.1"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rails_mind
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.3.1
 platform: ruby
 authors:
 - RailsMind