rails_mind 0.3.0 → 0.3.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 819a570e3f4f65e0fb8bae7de68b1a81ee744bccb77349858c640fcc912e95b4
-  data.tar.gz: 62df4fca70603e28458d277dcd8d734a5de04af7c514f997e15686122a02dbd2
+  metadata.gz: 9306b38b60c315db82ad49a0d338c625396362d4cb738f1b26639719c1805aab
+  data.tar.gz: 4a8b8e5e00a4b4141d8c95e0fcaf06cd1bb9b6e99980dc0c3a7248fb772c4dc8
 SHA512:
-  metadata.gz: 3583084684b6ac4b165864c14fbfe3922fc2f678c8d7aa9383418d95fdc528083bbe2327e5428d167d80e54a28445309eb6243bd3c6c3dac36fc8a67fa9e0630
-  data.tar.gz: e968233e4025ce470a83207214ef886032bff662faea00f824f3aceefad55319aa6d3235b62cc16368c5b8ccec7db86a6914ce1d5cb1a5db2c9bb282a543f0f2
+  metadata.gz: 5cae059308e349dc76067a47da37b21f2d11b6bc62b3690f5669187aa9663500aa3f4df0f39a4e1648489808263861223f12f82aaef7df161881d8c7f1710dd8
+  data.tar.gz: 5367481af0017d310b6b5de0bf82e29ecf4d734e366c03fbf2562033e71475eb976cd9d99f99d4c2634067350f22f078f6a2c0dbb760a5711f03c7833949fef6

data/CHANGELOG.md

@@ -1,5 +1,27 @@
 # Changelog
 
+## 0.3.2 — 2026-10-13
+
+Security fix for the scrubbing 0.3.1 turned on by default. Upgrade from 0.3.1.
+
+- An `Authorization` header in a message or string (`Bearer`, `Basic`, `Token` and other schemes; any case; `:`, `=` or `=>`; quoted, unquoted or wrapped in an array as `Net::HTTP#to_hash` prints it; `Proxy-Authorization` too) loses its credential. A `Digest` or `AWS4-HMAC-SHA256` header loses its whole value. 0.3.1 replaced only the scheme word and sent the credential after it.
+- Userinfo goes from a URL of any scheme (`redis://:password@`, `rediss://`, `amqp://`, `postgres://`, `mysql2://`, `mongodb+srv://` and others), not only `http(s)://`, up to the last `@` before the path, so an unencoded `@` in a password leaves nothing behind.
+- Provider credentials go without a key word next to them: GitHub tokens (`ghp_`, `gho_`, `ghu_`, `ghs_`, `ghr_`, `github_pat_`), AWS access key IDs (`AKIA`, `ASIA`), Stripe secret and restricted keys (`sk_live_`, `sk_test_`, `rk_…`), Slack tokens (`xox…-`), `sk-` API keys and JWTs.
+- Card numbers (13 to 19 digits, grouped by spaces or dashes or not, passing the Luhn check, also next to an expiry or another number) and US social security numbers (`123-45-6789`) go from free text: messages, backtraces, event names and properties. Identity, correlation and release values (`user_id`, `account_id`, `visitor_id`, `request_id`, `release` and the like) are left alone, so a numeric account ID is never mistaken for a card. In printed hashes, `phone` and `ssn` values go too.
+- Scrubbing reads at most 8 KB of a string, cut at whitespace, and every pass is linear, so a hostile message cannot stall the request that raised it. What is sent is still at most 1024 characters.
+- A PostgreSQL literal ending in a backslash no longer lets the next literal through, in query shapes or database error messages. Rails runs PostgreSQL with `standard_conforming_strings`, where `\` escapes a quote only inside `E'…'`, while MySQL lets it escape one; a statement or message containing a backslash is now read both ways and anything either reading takes for a literal goes. Statements without a backslash keep their fingerprints; one with an ambiguous `\'` keeps its text only up to that literal, so its fingerprint changes.
+- Tests: an adversarial corpus of these inputs that no secret may survive, as a message, an event property or a query shape.
+- `rails_mind:doctor` now exits 1 and names `RAILS_MIND_KEY` or the endpoint when collection would be off.
+
+## 0.3.1 — 2026-10-11
+
+- Send exception messages by default, scrubbed like every other string (emails, credentials, URL queries), and further for what generic scrubbing misses: database errors lose the failing row, key values, rejected input and quoted SQL literals, and printed hashes or keyword arguments lose the values of sensitive keys. `config.capture_exception_message = false` restores the placeholder. The flag is still not validated, so an app that assigned it from ENV boots as before.
+- Send the shape of each repeated query next to its fingerprint: the normalized SQL with literals replaced by `?`, comments removed, and at most 400 characters and 900 bytes, only for fingerprints that repeated within the request. One pass handles doubled and backslash-escaped quotes, `E''` and `$$` strings, hex numbers, comments containing quotes, a literal left open where a long statement is cut, and MySQL's double-quoted strings. `config.capture_query_shapes = false` sends fingerprints only.
+- Strip SQL comments (query log tags, Marginalia) before fingerprinting, so a query tagged per request keeps one fingerprint.
+- Name where each exception happened: `role` (`web`, `job`, `runner` or `console`), with the controller action as `route` for requests and `job_class` and `queue` for jobs, including unhandled job failures.
+- Keep the origin of library frames in backtraces (`gems/activerecord-8.1.4/lib/…`, `ruby/4.0.0/…`, `ruby/vendor_ruby/…`) so they can no longer pass for the application's own `lib/` files; an application directory that happens to be called `gems/` stays the application's.
+- Stop fingerprinting session settings and transaction control (`SET`, `SHOW`, `RESET`, `DISCARD`, `BEGIN`, `COMMIT`, `ROLLBACK`, `SAVEPOINT`, `RELEASE`, `PRAGMA` and the like), so a per-query `SET LOCAL statement_timeout` no longer reads as an N+1. They still count toward `query_count`.
+
 ## 0.3.0 — 2026-10-10
 
 - Enable bounded automatic endpoint sampling for routine successful requests, driven by expiring hosted policies. Preserve slow, failed, repeated-query requests and all other signal kinds from SDK sampling.

data/README.md

@@ -19,7 +19,6 @@ and [release notes](CHANGELOG.md).
 bundle install
 bin/rails generate rails_mind:install --plan
 bin/rails generate rails_mind:install
-bin/rails rails_mind:doctor
 ```
 
 Set **`RAILS_MIND_KEY`** for the destination application environment. It is the only
@@ -29,7 +28,14 @@ app's deployed `REVISION` file; telemetry still works when none is available.
 `RAILS_MIND_ENDPOINT` and `RAILS_MIND_RELEASE` are optional overrides. See
 [configuration and release detection](docs/sdk.md#configuration-and-release-detection).
 
-After setting the key, run `bin/rails rails_mind:verify` on the app's
+With the key in the app's environment, check the effective settings. Until the
+key is set, doctor names it and exits non-zero.
+
+```sh
+bin/rails rails_mind:doctor
+```
+
+Next, run `bin/rails rails_mind:verify` on the app's
 server. Match its check ID in RailsMind **Setup** and wait for **Check processed**.
 Then visit the app to confirm real traffic. This command sends one labeled
 diagnostic check, consumes one event of quota, and leaves business/performance

data/docs/sdk.md

@@ -16,6 +16,13 @@ gem "rails_mind", "~> 0.3.0"
 bundle install
 bin/rails generate rails_mind:install --plan
 bin/rails generate rails_mind:install
+```
+
+Then set `RAILS_MIND_KEY` ([configuration](#configuration-and-release-detection)) and, with
+it in the app's environment, check the effective settings. Until the key is set,
+doctor names it and exits non-zero.
+
+```sh
 bin/rails rails_mind:doctor
 ```
 
@@ -66,7 +73,9 @@ When upgrading an existing generated initializer, remove the old endpoint/releas
 assignments that directly read `ENV` so they do not overwrite the new defaults
 with `nil` or bypass platform detection. The installer preserves existing files.
 The doctor reports whether the effective endpoint, key, and release are present,
-without printing credential values or contacting the server.
+without printing credential values or contacting the server. While the key is
+missing or the endpoint is blank, it names the setting after the report and
+exits non-zero, as `rails_mind:verify` does.
 
 ## Verify the connection
 
@@ -180,7 +189,7 @@ RailsMind::Integrations::OpenTelemetry.attach!
 
 Attachment adds one idempotent processor; it does not replace the global provider, sampler, propagator, instrumentation, or other exporters. Its exporter only sanitizes and enqueues locally; the shared worker performs network I/O. Span context is attached at start so identity survives later export. Known safe attributes are allowlisted. Raw span names, exception span events, HTTP bodies/headers, SQL statements, and bind values are not exported to RailsMind. Existing third-party exporters retain their own behavior and privacy settings. [OpenTelemetry exporters](https://opentelemetry.io/docs/languages/ruby/exporters/).
 
-Request measurements and error reports come independently from Rails notifications/error reporting, even when detailed traces are sampled out. Routine requests separately follow the automatic endpoint sampling policy described below. Request measurements cover controller actions, not assets/health checks rejected before ActionController or streaming-body completion. Repeated-query evidence contains only normalized-query hashes and counts; it is a suspicion, not proof of an N+1. Hash normalization handles common literals, not every SQL grammar. Use Bullet/Prosopite or an explicit query-bound test in development to verify fixes. No production object scans are installed.
+Request measurements and error reports come independently from Rails notifications/error reporting, even when detailed traces are sampled out. Routine requests separately follow the automatic endpoint sampling policy described below. Request measurements cover controller actions, not assets/health checks rejected before ActionController or streaming-body completion. Repeated-query evidence contains normalized-query hashes and counts, plus the shape of each query that repeated: the normalized SQL with string and numeric literals replaced by `?`, comments removed, and at most 400 characters. Set `config.capture_query_shapes = false` to send hashes only. Session settings and transaction control (`SET`, `SHOW`, `BEGIN`, `COMMIT`, `SAVEPOINT` and the like) count toward `query_count` but are never fingerprinted. It is a suspicion, not proof of an N+1. Normalization handles common literals, not every SQL grammar. Use Bullet/Prosopite or an explicit query-bound test in development to verify fixes. No production object scans are installed.
 
 Full metrics/logs SDK integration is deferred. Hosted request aggregates describe retained events; sampling can bias their distributions. The SDK does not capture application logs.
 
@@ -328,7 +337,7 @@ RailsMind.flush(timeout: 2)
 
 These are per-process counters. `dropped` counts known local discards, such as queue overflow or oversize payloads. `delivery_unconfirmed` counts events whose delivery attempts ended without a success response, including interrupted in-flight batches; some may have been accepted by the server. `delivered` counts submitted events in batches with a successful HTTP response; it does not prove server retention or background processing. The doctor prints stats for its process; inspect running app stats through your existing operational tooling. Only the routine-request report above sends queue-loss observations to the hosted app. All signal kinds can still undercount after crashes, rejected payloads, hooks, or outages. This is best-effort monitoring, not an accounting ledger; use a transactional outbox when durable business-event delivery is required.
 
-Redaction occurs before buffering and again after custom `before_send`. Sensitive property names are filtered, nesting/string/collection sizes are capped, emails and common credential forms are scrubbed, exception messages are omitted by default, and backtraces drop machine-home prefixes. Bodies, cookies, credentials, SQL binds, and job arguments are never selected by integrations. Arbitrary user-provided text may contain sensitive content that generic redaction cannot recognize: use small, explicitly reviewed business properties. Opting into exception messages requires reviewing `capture_exception_message` and your custom scrubbing policy.
+Redaction occurs before buffering and again after custom `before_send`. Sensitive property names are filtered, nesting/string/collection sizes are capped, emails and common credential forms are scrubbed (exception messages included), and backtraces drop machine-home prefixes while keeping a `gems/<name>-<version>/` or `ruby/<version>/` origin for library frames. Error events also name where they happened: `role`, plus the controller action as `route` or the job's `job_class` and `queue`. Bodies, cookies, credentials, SQL binds, and job arguments are never selected by integrations. Arbitrary user-provided text may contain sensitive content that generic redaction cannot recognize: use small, explicitly reviewed business properties. Exception messages can contain whatever your code interpolated into them; review your scrubbing policy, or set `config.capture_exception_message = false` to replace every message with a placeholder before it leaves.
 
 The final local synthetic burst benchmark on Ruby 4.0.6 processed 10,000 calls in 1.7612 seconds (176.12 μs/call), delivering all 10,000 with zero drops/retries at the 1,000-event queue bound. This measures local sanitization/enqueue with a no-op transport, not production request overhead or network delivery. Earlier runs did overflow; thread scheduling and downstream speed affect the result. Deterministic tests separately verify overflow losses and bounded shutdown during a hung transport. Run `ruby script_benchmark.rb` on the deployment hardware before setting a production budget.
 

data/lib/generators/rails_mind/install/templates/initializer.rb.tt

@@ -14,8 +14,12 @@ RailsMind.configure do |config|
   # config.analytics = false
   # config.flags = false
 
-  # Exception messages are omitted by default. Review before enabling.
-  # config.capture_exception_message = true
+  # Exception messages are scrubbed (emails, credentials) and sent. Set false to replace
+  # them with a placeholder and keep them in your app.
+  # config.capture_exception_message = false
+  # Repeated queries are sent as a fingerprint plus their shape (the normalized SQL with
+  # literals replaced and comments removed). Set false to send the fingerprints only.
+  # config.capture_query_shapes = false
   # config.before_send = ->(event) { event } # Return nil to drop an event.
 end
 

data/lib/rails_mind/client.rb

@@ -40,7 +40,9 @@ module RailsMind
       false
     end
 
-    def capture_exception(error, handled: false, severity: :error, context: {}, source: nil, **_options)
+    LOCATION_KEYS = %i[role route job_class queue].freeze
+
+    def capture_exception(error, handled: false, severity: :error, context: {}, source: nil, location: nil, **_options)
       return false unless config.errors
       # Rails reports an exception through more than one boundary. A weak key map
       # avoids retaining exceptions; identity dedup applies only to this object.
@@ -48,13 +50,14 @@ module RailsMind
         return false if @reported_errors.key?(error)
         @reported_errors[error] = true
       end
-      message = config.capture_exception_message ? @redactor.scrub(error.message.to_s) : "[Message omitted]"
+      message = config.capture_exception_message ? @redactor.scrub_message(error) : "[Message omitted]"
       frames = Array(error.backtrace).first(40).map { |frame| safe_frame(frame) }
       Context.with(context.slice(*Context::KEYS)) do
+        where = (location || {}).slice(*LOCATION_KEYS).select { |_, value| value.is_a?(String) && !value.empty? }.transform_values { |value| value[0, 200] }
         emit(:error, error.class.name, status: "error", properties: {
           exception_class: error.class.name, message: message, backtrace: frames,
           handled: handled, severity: severity.to_s, source: source
-        })
+        }.merge(where))
       end
     rescue StandardError
       false
@@ -104,12 +107,27 @@ module RailsMind
       time.iso8601(6)
     end
 
+    # An installed gem or Ruby's own library keeps its origin ("gems/rack-3.1.8/lib/…",
+    # "ruby/4.0.0/…") so it can never pass for the application's lib/. A gem
+    # directory is name-version, or name-sha for a Bundler git checkout.
+    GEM_FRAME = %r{\A.*/gems/([^/]+-(?:\d[^/]*|\h{7,40}))/}
+    RUBY_FRAME = %r{\A.*?/lib/ruby/((?:site_ruby/|vendor_ruby/)?\d[^/]*)/}
+    APP_RELATIVE = %r{\A(?:app|lib|config|test|spec)/}
+
     def safe_frame(frame)
       frame = frame.to_s
       frame = frame.delete_prefix("#{Rails.root}/") if defined?(Rails) && Rails.respond_to?(:root) && Rails.root
-      # Remove machine/user home paths, retaining source filename + line + method.
-      frame = frame.sub(%r{\A.*?/(app|lib|config|test|spec)/}, '\\1/')
-      frame = frame.sub(%r{\A/(?:[^/:]+/)*([^/:]+:\d+)}, '\\1')
+      if frame.match?(APP_RELATIVE)
+        # Already the application's own path.
+      elsif frame.match?(GEM_FRAME)
+        frame = frame.sub(GEM_FRAME, 'gems/\\1/')
+      elsif frame.match?(RUBY_FRAME)
+        frame = frame.sub(RUBY_FRAME, 'ruby/\\1/')
+      else
+        # Remove machine/user home paths, retaining source filename + line + method.
+        frame = frame.sub(%r{\A.*?/(app|lib|config|test|spec)/}, '\\1/')
+        frame = frame.sub(%r{\A/(?:[^/:]+/)*([^/:]+:\d+)}, '\\1')
+      end
       @redactor.scrub(frame)
     end
 

data/lib/rails_mind/configuration.rb

@@ -9,7 +9,7 @@ module RailsMind
     attr_accessor :endpoint, :token, :release, :apm, :errors, :analytics, :flags,
       :queue_capacity, :batch_size, :max_batch_bytes, :max_event_bytes, :flush_interval,
       :max_retries, :retry_base, :open_timeout, :read_timeout, :shutdown_timeout,
-      :before_send, :capture_exception_message, :automatic_sampling
+      :before_send, :capture_exception_message, :capture_query_shapes, :automatic_sampling
 
     def initialize(env: ENV, root: nil)
       @endpoint = nonblank(env["RAILS_MIND_ENDPOINT"]) || DEFAULT_ENDPOINT
@@ -26,7 +26,8 @@ module RailsMind
       @open_timeout = 1.0
       @read_timeout = 2.0
       @shutdown_timeout = 2.0
-      @capture_exception_message = false
+      @capture_exception_message = true
+      @capture_query_shapes = true
       @automatic_sampling = true
     end
 
@@ -34,7 +35,12 @@ module RailsMind
 
     def validate!
       return self unless enabled?
-      raise ArgumentError, "automatic_sampling must be true or false" unless automatic_sampling == true || automatic_sampling == false
+      # capture_exception_message stays unvalidated as in 0.3.0, so an app that
+      # assigned it from ENV still boots after upgrading.
+      %i[automatic_sampling capture_query_shapes].each do |flag|
+        value = public_send(flag)
+        raise ArgumentError, "#{flag} must be true or false" unless value == true || value == false
+      end
       uri = URI(endpoint)
       loopback = %w[localhost 127.0.0.1 ::1].include?(uri.hostname)
       raise ArgumentError, "endpoint must use HTTPS (HTTP only on loopback)" unless uri.is_a?(URI::HTTPS) || (uri.is_a?(URI::HTTP) && loopback)

data/lib/rails_mind/diagnostic.rb

@@ -36,5 +36,17 @@ module RailsMind
           "No network check is performed and credential values are never printed." ]
       }
     end
+
+    # What keeps collection off, one sentence each, naming the setting to change.
+    def problems
+      [].tap do |found|
+        if @config.token.to_s.strip.empty?
+          found << "RAILS_MIND_KEY is not set, so collection is off. Set it in this app's environment to the ingest key from RailsMind Setup, restart, then run bin/rails rails_mind:verify."
+        end
+        if @config.endpoint.to_s.strip.empty?
+          found << "The RailsMind endpoint is blank, so collection is off. Leave config.endpoint and RAILS_MIND_ENDPOINT unset to use #{Configuration::DEFAULT_ENDPOINT}, or set your RailsMind instance's URL."
+        end
+      end
+    end
   end
 end

data/lib/rails_mind/integrations/rails.rb

@@ -1,11 +1,38 @@
 require "active_support/notifications"
+require "active_support/inflector"
 require "digest"
 
 module RailsMind
   module Integrations
+    # Where an exception happened, from the execution context Rails hands its
+    # error subscribers: the controller action, the job, or the kind of process.
+    # A runner script wins over a job it ran inline, whose context Rails leaves set.
+    def self.location(context, source)
+      controller, job = context[:controller], context[:job]
+      if source == "application.runner.railties"
+        { role: "runner" }
+      elsif controller.respond_to?(:action_name)
+        { role: "web", route: "#{controller.class.name}##{controller.action_name}" }
+      elsif job.respond_to?(:queue_name)
+        { role: "job", job_class: job.class.name, queue: job.queue_name.to_s }
+      elsif defined?(::Rails::Console)
+        { role: "console" }
+      else
+        {}
+      end
+    end
+
+    # The routed action of a request, named the way request events name it.
+    def self.request_location(env)
+      params = env["action_dispatch.request.path_parameters"]
+      return { role: "web" } unless params.is_a?(Hash) && params[:controller] && params[:action]
+      { role: "web", route: "#{ActiveSupport::Inflector.camelize(params[:controller].to_s)}Controller##{params[:action]}" }
+    end
+
     class ErrorSubscriber
       def report(error, handled:, severity:, context:, source: nil, **options)
-        RailsMind.capture_exception(error, handled: handled, severity: severity, context: context, source: source)
+        RailsMind.capture_exception(error, handled: handled, severity: severity, context: context, source: source,
+          location: Integrations.location(context, source))
       end
     end
 
@@ -16,12 +43,12 @@ module RailsMind
         Context.with(request_id: env["action_dispatch.request_id"] || SecureRandom.uuid) do
           Context.with_trace do
             previous = Thread.current[:rails_mind_queries]
-            Thread.current[:rails_mind_queries] = { count: 0, fingerprints: Hash.new(0) }
+            Thread.current[:rails_mind_queries] = { count: 0, fingerprints: Hash.new(0), shapes: {} }
             @app.call(env)
           rescue StandardError => error
             # Capture identity before this request boundary unwinds. Object
             # dedup still leaves other error reporters unaffected.
-            RailsMind.capture_exception(error, source: "rack")
+            RailsMind.capture_exception(error, source: "rack", location: Integrations.request_location(env))
             raise
           ensure
             Thread.current[:rails_mind_queries] = previous
@@ -74,6 +101,84 @@ module RailsMind
     end
 
     module RailsNotifications
+      SHAPE_CHARS = 400
+      # The server keeps a shape of at most 1000 bytes; redaction can lengthen one slightly.
+      SHAPE_BYTES = 900
+      # Session settings and transaction control are bookkeeping, not reads: a
+      # per-query `SET LOCAL statement_timeout` must never look like an N+1.
+      BOOKKEEPING = /\A(?:SET|SHOW|RESET|DISCARD|BEGIN|START TRANSACTION|COMMIT|END|ABORT|ROLLBACK|SAVEPOINT|RELEASE|PRAGMA)\b/i
+      # One pass over the statement, so a quote inside a comment or a comment
+      # marker inside a string cannot derail it: strings (doubled or backslash-
+      # escaped quotes, E'' and $$ forms, and one left open where the statement
+      # was cut), numbers and hex literals become ?, and comments go. Double
+      # quotes are identifiers, except on MySQL, where they can be strings.
+      LEXEME = /(?<!\w)[Ee]'(?:[^'\\]|\\.|'')*'?|'(?:[^'\\]|\\.|'')*'?|\$(\w*)\$.*?(?:\$\1\$|\z)|"(?:[^"\\]|\\.|"")*"?|`[^`]*`?|--[^\n]*|\/\*.*?(?:\*\/|\z)|\b0[xX]\h+\b|\b\d+(?:\.\d+)?(?:[eE][-+]?\d+)?\b/m
+      # The same lexemes where a backslash escapes nothing outside E'...', as on
+      # PostgreSQL (Rails turns standard_conforming_strings on) and SQLite. Which
+      # rule applies is not always known (MySQL can turn backslash escapes off),
+      # so a statement with a backslash is read both ways and anything either
+      # reading takes for a literal goes. Without a backslash both agree.
+      STANDARD_LEXEME = /(?<!\w)[Ee]'(?:[^'\\]|\\.|'')*'?|'(?:[^']|'')*'?|\$(\w*)\$.*?(?:\$\1\$|\z)|"(?:[^"]|"")*"?|`[^`]*`?|--[^\n]*|\/\*.*?(?:\*\/|\z)|\b0[xX]\h+\b|\b\d+(?:\.\d+)?(?:[eE][-+]?\d+)?\b/m
+
+      def self.normalize(sql, mysql: false)
+        sql = sql.scrub("?")
+        text = sql.include?("\\") ? read_both_ways(sql, mysql) : sql.gsub(LEXEME) { |token| replacement(token, mysql) }
+        text.gsub(/\s+/, " ").strip
+      end
+
+      def self.replacement(token, mysql)
+        case token[0]
+        when "-", "/" then " "
+        when "`" then token
+        when '"' then mysql ? "?" : token
+        else "?"
+        end
+      end
+
+      # Literal spans from either reading merge into one ?; a comment that
+      # overlaps one joins it. Kept identifiers stay only where no reading saw a
+      # literal.
+      def self.read_both_ways(sql, mysql)
+        literals, comments = [], []
+        [ STANDARD_LEXEME, LEXEME ].each do |pattern|
+          sql.scan(pattern) do
+            token = Regexp.last_match
+            case replacement(token[0], mysql)
+            when "?" then literals << token.offset(0)
+            when " " then comments << token.offset(0)
+            end
+          end
+        end
+        literals = merge(literals)
+        loop do
+          # Merged literal spans are sorted and disjoint, so their ends are too.
+          joining, comments = comments.partition do |start, finish|
+            overlap = literals.bsearch { |_, last| last > start }
+            overlap && overlap[0] < finish
+          end
+          break if joining.empty?
+          literals = merge(literals + joining)
+        end
+        spans = (literals.map { |span| span + [ "?" ] } + merge(comments).map { |span| span + [ " " ] }).sort
+        output, position = +"", 0
+        spans.each do |start, finish, mark|
+          output << sql[position...start] << mark
+          position = finish
+        end
+        output << sql[position..]
+      end
+
+      def self.merge(spans)
+        spans.sort.each_with_object([]) do |(start, finish), merged|
+          if merged.any? && start < merged.last[1]
+            merged.last[1] = [ merged.last[1], finish ].max
+          else
+            merged << [ start, finish ]
+          end
+        end
+      end
+      private_class_method :replacement, :read_both_ways, :merge
+
       class << self
         def install!
           return if @subscriptions
@@ -83,20 +188,33 @@ module RailsMind
             queries = Thread.current[:rails_mind_queries]
             counts = queries&.fetch(:fingerprints, {}) || {}
             status = payload[:status] || (payload[:exception] ? 500 : 200)
+            top = counts.sort_by { |_, count| -count }.first(10).to_h
+            properties = { controller: payload[:controller], action: payload[:action],
+              method: payload[:method], view_runtime: payload[:view_runtime], db_runtime: payload[:db_runtime],
+              query_count: queries&.fetch(:count), repeated_query_count: counts.values.sum { |n| [ n - 1, 0 ].max },
+              query_fingerprints: top }
+            # Shapes go only for the queries that repeated: that is what a finding is about.
+            if RailsMind.client.config.capture_query_shapes
+              shapes = queries&.fetch(:shapes, {}) || {}
+              properties[:query_shapes] = top.filter_map { |fingerprint, count| [ fingerprint, shapes[fingerprint] ] if count >= 2 && shapes[fingerprint] }.to_h
+            end
             RailsMind.client.emit(:request, "#{payload[:controller]}##{payload[:action]}", duration_ms: event.duration,
-              status: status.to_s, properties: { controller: payload[:controller], action: payload[:action],
-                method: payload[:method], view_runtime: payload[:view_runtime], db_runtime: payload[:db_runtime],
-                query_count: queries&.fetch(:count), repeated_query_count: counts.values.sum { |n| [ n - 1, 0 ].max },
-                query_fingerprints: counts.sort_by { |_, count| -count }.first(10).to_h })
+              status: status.to_s, properties: properties)
           end
           subscribe("sql.active_record") do |event|
             queries = Thread.current[:rails_mind_queries]
             next unless queries && !event.payload[:cached] && !%w[SCHEMA TRANSACTION].include?(event.payload[:name])
             queries[:count] += 1
-            # Normalize locally, transmit only one-way hashes; never SQL or binds.
-            normalized = event.payload[:sql].to_s[0, 16_384].gsub(/'(?:[^']|'')*'/, "?").gsub(/\b\d+\b/, "?").gsub(/\s+/, " ")
+            # Comments (query log tags, Marginalia) and literals go before hashing, so one
+            # query has one fingerprint. Binds are never read. The shape is this same
+            # normalized text, capped, and leaves only when capture_query_shapes allows.
+            mysql = event.payload[:connection].respond_to?(:adapter_name) && event.payload[:connection].adapter_name.to_s.match?(/mysql|trilogy/i)
+            normalized = RailsNotifications.normalize(event.payload[:sql].to_s[0, 16_384], mysql: mysql)
+            next if normalized.match?(BOOKKEEPING)
             fingerprint = Digest::SHA256.hexdigest(normalized)[0, 24]
-            queries[:fingerprints][fingerprint] += 1 if queries[:fingerprints].key?(fingerprint) || queries[:fingerprints].size < 100
+            next unless queries[:fingerprints].key?(fingerprint) || queries[:fingerprints].size < 100
+            queries[:fingerprints][fingerprint] += 1
+            (queries[:shapes] ||= {})[fingerprint] ||= normalized[0, SHAPE_CHARS].byteslice(0, SHAPE_BYTES).scrub("") if RailsMind.client.config.capture_query_shapes
           end
           %w[enqueue.active_job enqueue_at.active_job perform_start.active_job perform.active_job enqueue_retry.active_job retry_stopped.active_job discard.active_job].each do |name|
             subscribe(name) do |event|
@@ -128,7 +246,10 @@ module RailsMind
                   kind = enqueue || operation == "perform_start" ? :span : :job
                   RailsMind.client.emit(kind, job.class.name, duration_ms: event.duration, status: status,
                     properties: job_properties(job, operation))
-                  RailsMind.capture_exception(error, source: "active_job") if error
+                  if error
+                    RailsMind.capture_exception(error, source: "active_job",
+                      location: { role: "job", job_class: job.class.name, queue: job.queue_name.to_s })
+                  end
                 end
               end
             end

data/lib/rails_mind/redactor.rb

@@ -2,13 +2,22 @@ module RailsMind
   class Redactor
     FILTER = /password|passwd|secret|token|api[_-]?key|access[_-]?key|private[_-]?key|authorization|cookie|credential|session|email|phone|address|credit|card|body|params|arguments|bind|sql|db\.statement|query\.text/i
     MAX_STRING = 1024
+    # Scrubbing reads at most this much of a string; what it sends is still at
+    # most MAX_STRING. Every pass below is linear, and the cap bounds the rest.
+    INPUT_LIMIT = 8192
+    # Identity, correlation and release values are not free text: a numeric
+    # account ID that passes the Luhn check is not a card number.
+    PLAIN_FIELDS = %w[id kind occurred_at status release trace_id span_id parent_span_id request_id job_id visitor_id user_id account_id].freeze
 
     def call(value, depth = 0)
       return "[TRUNCATED]" if depth > 5
       case value
       when Hash
         value.first(50).to_h.transform_keys { |key| key.to_s[0, 100] }.each_with_object({}) do |(key, item), clean|
-          clean[key] = key.match?(FILTER) ? "[FILTERED]" : call(item, depth + 1)
+          clean[key] = if key.match?(FILTER) then "[FILTERED]"
+          elsif depth.zero? && PLAIN_FIELDS.include?(key) && (item.is_a?(String) || item.is_a?(Symbol)) then scrub(item.to_s, free_text: false)
+          else call(item, depth + 1)
+          end
         end
       when Array then value.first(50).map { |item| call(item, depth + 1) }
       when String then scrub(value)
@@ -19,14 +28,141 @@ module RailsMind
       end
     end
 
-    def scrub(value)
-      value.encode("UTF-8", invalid: :replace, undef: :replace, replace: "�")
-        .gsub(/\b[A-Z0-9._%+\-]+@[A-Z0-9.\-]+\.[A-Z]{2,}\b/i, "[EMAIL]")
+    # An authorization scheme names the credential that follows it, so the
+    # scheme and the credential go together. Digest and AWS signature headers
+    # carry several credential parts, so their whole value goes.
+    SCHEME = "(?:bearer|basic|token|negotiate|ntlm|bot|apikey)\\s+".freeze
+    WHOLE_VALUE_SCHEME = "(?:digest|aws4-hmac-sha256)\\s[^\\n]*".freeze
+
+    # Exception messages carry what generic scrubbing cannot recognize: database
+    # errors quote the row, key or input that failed (and the SQL with its
+    # literals), and Ruby prints hashes and keyword arguments with their values,
+    # header values wrapped in arrays (Net::HTTP#to_hash) included.
+    DATABASE_ERROR = /\A(?:PG|Mysql2|Trilogy|SQLite3)::|StatementInvalid|RecordNotUnique|NotNullViolation|ForeignKeyViolation|ValueTooLong|RangeError\z/
+    SENSITIVE_VALUE = /((?:"|:)?(?:password|passwd|secret|token|api[_-]?key|access[_-]?key|private[_-]?key|authorization|credential|cookie|session|phone|ssn)\w{0,64}"?\s*(?:=>|:)\s*(?:\[(?!FILTERED\]))?)(?:\[FILTERED\]|"(?:[^"\\]|\\.)*"|'(?:[^'\\]|\\.)*'|#{WHOLE_VALUE_SCHEME}|(?:#{SCHEME})?[^\s,}\])]+)/i
+
+    # PostgreSQL connections use standard_conforming_strings, where a backslash
+    # does not escape the quote of a '...' literal (it does inside E'...');
+    # MySQL lets it escape the quote. Text with a backslash is read both ways,
+    # and anything either reading takes for a literal goes.
+    STANDARD_LITERAL = /(?<!\w)[Ee]'(?:[^'\\]|\\.|'')*'?|'(?:[^']|'')*'?/
+    ESCAPED_LITERAL = /'(?:[^'\\]|\\.|'')*'?/
+
+    def scrub_message(error)
+      text = bounded(error.message.to_s.encode("UTF-8", invalid: :replace, undef: :replace, replace: "�"))
+      text = text.gsub(SENSITIVE_VALUE, '\\1"[FILTERED]"')
+        .gsub(/(Failing row contains )\([^\n]*\)/, "\\1(…)")
+        .gsub(/(Key \([^)\n]*\)=)\([^\n]*\)/, "\\1(…)")
+        .gsub(/(invalid input (?:syntax|value) for [^:\n]*: )"[^"\n]*"/, '\\1"…"')
+      text = mask_literals(text) if database_error?(error)
+      scrub(text)
+    end
+
+    # Userinfo runs to the last @ before the path, so an unencoded @ in a
+    # password cannot leave its tail behind.
+    USERINFO = %r{\b([a-z][a-z0-9+.\-]{0,31}://)[^\s/]*@}i
+    EMAIL = /\b[A-Z0-9._%+\-]{1,128}@[A-Z0-9.\-]{1,253}\.[A-Z]{2,}\b/i
+    AUTHORIZATION = /((?:proxy-)?authorization["']?\s*(?:=>|[=:])\s*(?:\[(?!FILTERED\]))?\s*["']?)(?:\[FILTERED\]|#{WHOLE_VALUE_SCHEME}|(?:#{SCHEME})?[^\s,;&"']+)/i
+    KEY_VALUE = /((?:password|secret|token|api[_-]?key|access[_-]?key|private[_-]?key|authorization)\s*(?:=>|[=:])\s*)(?:#{SCHEME})?[^\s,;&]+/i
+    PROVIDER_TOKEN = /\b(?:gh[pousr]_[A-Za-z0-9]{36,}|github_pat_\w{22,}|(?:AKIA|ASIA)[A-Z0-9]{16}\b|[sr]k_(?:live|test)_[A-Za-z0-9]{10,}|xox[abposr]-[A-Za-z0-9-]{10,}|sk-(?:proj-|ant-)?[A-Za-z0-9_-]{20,}|eyJ[\w-]{5,512}\.[\w-]{5,}\.[\w-]*)/
+    SSN = /\b\d{3}([- ])\d{2}\1\d{4}\b/
+    DIGIT_RUN = /(?<!\w)\d(?:[ -]?\d)*(?!\w)/
+
+    def scrub(value, free_text: true)
+      text = bounded(value.encode("UTF-8", invalid: :replace, undef: :replace, replace: "�"))
+        .gsub(USERINFO, '\\1[FILTERED]@')
+        .gsub(EMAIL, "[EMAIL]")
+        .gsub(AUTHORIZATION, '\\1[FILTERED]')
         .gsub(/\bBearer\s+\S+/i, "Bearer [FILTERED]")
-        .gsub(/((?:password|secret|token|api[_-]?key|access[_-]?key|private[_-]?key|authorization)\s*[=:]\s*)[^\s,;&]+/i, '\\1[FILTERED]')
-        .gsub(%r{(https?://)[^\s/@]+:[^\s/@]+@}, '\\1[FILTERED]@')
-        .gsub(%r{(https?://[^\s?#]+)[?#][^\s]*}, '\\1')
-        .slice(0, MAX_STRING)
+        .gsub(KEY_VALUE, '\\1[FILTERED]')
+      text = text.gsub(PROVIDER_TOKEN, "[FILTERED]").gsub(SSN, "[FILTERED]").gsub(DIGIT_RUN) { |run| mask_cards(run) } if free_text
+      text.gsub(%r{https?://\S+}) { |url| url[/\A[^?#]*/] }.slice(0, MAX_STRING)
+    end
+
+    private
+
+    def database_error?(error)
+      error.class.ancestors.any? { |ancestor| ancestor.name.to_s.match?(DATABASE_ERROR) }
+    end
+
+    # A cut string ends at its last whitespace, so no half token is left that a
+    # pass could no longer recognize.
+    def bounded(text)
+      return text if text.length <= INPUT_LIMIT
+      text = text[0, INPUT_LIMIT]
+      cut = text.rindex(/\s/)
+      cut ? text[0, cut] : text
+    end
+
+    def mask_literals(text)
+      return text.gsub(ESCAPED_LITERAL, "?") unless text.include?("\\")
+      spans = [ STANDARD_LITERAL, ESCAPED_LITERAL ].flat_map { |pattern| text.to_enum(:scan, pattern).map { Regexp.last_match.offset(0) } }
+      output, position = +"", 0
+      merge(spans).each do |start, finish|
+        output << text[position...start] << "?"
+        position = finish
+      end
+      output << text[position..]
+    end
+
+    def merge(spans)
+      spans.sort.each_with_object([]) do |(start, finish), merged|
+        if merged.any? && start < merged.last[1]
+          merged.last[1] = [ merged.last[1], finish ].max
+        else
+          merged << [ start, finish ]
+        end
+      end
+    end
+
+    # A card number is 13 to 19 digits, maybe in groups split by spaces or
+    # dashes, that pass the Luhn check. A run of digit groups can hold one next
+    # to other numbers (an expiry, a quantity), so every span of whole groups
+    # with at most 19 digits is tried, and the longest that passes goes.
+    def mask_cards(run)
+      groups = run.to_enum(:scan, /\d+/).map { Regexp.last_match.offset(0) }
+      digits = run.delete(" -")
+      return run if digits.length < 13
+      bounds, count = [], 0
+      groups.each { |start, finish| bounds << [ count, count += finish - start ] }
+      plain, doubled = luhn_sums(digits)
+      output, position, first = +"", 0, 0
+      while first < groups.size
+        longest, from = nil, bounds[first][0]
+        (first...groups.size).each do |last|
+          to = bounds[last][1]
+          break if to - from > 19
+          next if to - from < 13
+          parity = to % 2
+          sum = doubled[parity][to] - doubled[parity][from] + plain[1 - parity][to] - plain[1 - parity][from]
+          longest = last if (sum % 10).zero?
+        end
+        if longest
+          output << run[position...groups[first][0]] << "[FILTERED]"
+          position, first = groups[longest][1], longest + 1
+        else
+          first += 1
+        end
+      end
+      output << run[position..]
+    end
+
+    # Prefix sums of each digit, plain and Luhn-doubled, split by position
+    # parity. In a span ending at index `to`, Luhn doubles the digits whose
+    # position has the parity of `to`, so any span's checksum is four lookups.
+    def luhn_sums(digits)
+      plain, doubled = Array.new(2) { [ 0 ] }, Array.new(2) { [ 0 ] }
+      digits.each_char.with_index do |char, index|
+        value = char.ord - 48
+        twice = value * 2
+        twice -= 9 if twice > 9
+        [ 0, 1 ].each do |parity|
+          own = index % 2 == parity
+          plain[parity] << plain[parity].last + (own ? value : 0)
+          doubled[parity] << doubled[parity].last + (own ? twice : 0)
+        end
+      end
+      [ plain, doubled ]
     end
   end
 end

data/lib/rails_mind/version.rb

@@ -1,3 +1,3 @@
 module RailsMind
-  VERSION = "0.3.0"
+  VERSION = "0.3.2"
 end

data/lib/tasks/rails_mind.rake

@@ -7,9 +7,15 @@ namespace :rails_mind do
     exit 1 unless result[:success]
   end
 
-  desc "Inspect dependencies and report the non-destructive integration plan"
+  desc "Inspect dependencies and report the non-destructive integration plan; fails while collection is off"
   task doctor: :environment do
     require "rails_mind/diagnostic"
-    puts JSON.pretty_generate(RailsMind::Diagnostic.new(root: Rails.root).report.merge(collector: RailsMind.stats))
+    diagnostic = RailsMind::Diagnostic.new(root: Rails.root)
+    puts JSON.pretty_generate(diagnostic.report.merge(collector: RailsMind.stats))
+    problems = diagnostic.problems
+    unless problems.empty?
+      warn problems
+      exit 1
+    end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rails_mind
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.3.2
 platform: ruby
 authors:
 - RailsMind