rails_mind 0.3.1 → 0.3.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 902c75e77dcb5d76b75de68e62e73fe7a92f38ec05f621084bc6a61c047cb186
-  data.tar.gz: b499762228b099f7d940a3b4aa58ba1d1c73d4a9c53958a02e379982fa08b28b
+  metadata.gz: 9306b38b60c315db82ad49a0d338c625396362d4cb738f1b26639719c1805aab
+  data.tar.gz: 4a8b8e5e00a4b4141d8c95e0fcaf06cd1bb9b6e99980dc0c3a7248fb772c4dc8
 SHA512:
-  metadata.gz: ae733243113a0206980e2b34f2545f5648c00b4e032d32ae2727bd8b5da85d67067970cc73ff1654eef1eb93782457d71cb65c45e9a6bfcde5f6f7fc03985310
-  data.tar.gz: 48d196c72e82519469bc6afba1e47a585012a9ff3f0812c243ddd9b00378f6512f5b7d4f8d62fbd5158a029e2166d35e8d3373588fc3458c87c324848d477ed2
+  metadata.gz: 5cae059308e349dc76067a47da37b21f2d11b6bc62b3690f5669187aa9663500aa3f4df0f39a4e1648489808263861223f12f82aaef7df161881d8c7f1710dd8
+  data.tar.gz: 5367481af0017d310b6b5de0bf82e29ecf4d734e366c03fbf2562033e71475eb976cd9d99f99d4c2634067350f22f078f6a2c0dbb760a5711f03c7833949fef6

data/CHANGELOG.md

@@ -1,6 +1,19 @@
 # Changelog
 
-## 0.3.1 — unreleased
+## 0.3.2 — 2026-10-13
+
+Security fix for the scrubbing 0.3.1 turned on by default. Upgrade from 0.3.1.
+
+- An `Authorization` header in a message or string (`Bearer`, `Basic`, `Token` and other schemes; any case; `:`, `=` or `=>`; quoted, unquoted or wrapped in an array as `Net::HTTP#to_hash` prints it; `Proxy-Authorization` too) loses its credential. A `Digest` or `AWS4-HMAC-SHA256` header loses its whole value. 0.3.1 replaced only the scheme word and sent the credential after it.
+- Userinfo goes from a URL of any scheme (`redis://:password@`, `rediss://`, `amqp://`, `postgres://`, `mysql2://`, `mongodb+srv://` and others), not only `http(s)://`, up to the last `@` before the path, so an unencoded `@` in a password leaves nothing behind.
+- Provider credentials go without a key word next to them: GitHub tokens (`ghp_`, `gho_`, `ghu_`, `ghs_`, `ghr_`, `github_pat_`), AWS access key IDs (`AKIA`, `ASIA`), Stripe secret and restricted keys (`sk_live_`, `sk_test_`, `rk_…`), Slack tokens (`xox…-`), `sk-` API keys and JWTs.
+- Card numbers (13 to 19 digits, grouped by spaces or dashes or not, passing the Luhn check, also next to an expiry or another number) and US social security numbers (`123-45-6789`) go from free text: messages, backtraces, event names and properties. Identity, correlation and release values (`user_id`, `account_id`, `visitor_id`, `request_id`, `release` and the like) are left alone, so a numeric account ID is never mistaken for a card. In printed hashes, `phone` and `ssn` values go too.
+- Scrubbing reads at most 8 KB of a string, cut at whitespace, and every pass is linear, so a hostile message cannot stall the request that raised it. What is sent is still at most 1024 characters.
+- A PostgreSQL literal ending in a backslash no longer lets the next literal through, in query shapes or database error messages. Rails runs PostgreSQL with `standard_conforming_strings`, where `\` escapes a quote only inside `E'…'`, while MySQL lets it escape one; a statement or message containing a backslash is now read both ways and anything either reading takes for a literal goes. Statements without a backslash keep their fingerprints; one with an ambiguous `\'` keeps its text only up to that literal, so its fingerprint changes.
+- Tests: an adversarial corpus of these inputs that no secret may survive, as a message, an event property or a query shape.
+- `rails_mind:doctor` now exits 1 and names `RAILS_MIND_KEY` or the endpoint when collection would be off.
+
+## 0.3.1 — 2026-10-11
 
 - Send exception messages by default, scrubbed like every other string (emails, credentials, URL queries), and further for what generic scrubbing misses: database errors lose the failing row, key values, rejected input and quoted SQL literals, and printed hashes or keyword arguments lose the values of sensitive keys. `config.capture_exception_message = false` restores the placeholder. The flag is still not validated, so an app that assigned it from ENV boots as before.
 - Send the shape of each repeated query next to its fingerprint: the normalized SQL with literals replaced by `?`, comments removed, and at most 400 characters and 900 bytes, only for fingerprints that repeated within the request. One pass handles doubled and backslash-escaped quotes, `E''` and `$$` strings, hex numbers, comments containing quotes, a literal left open where a long statement is cut, and MySQL's double-quoted strings. `config.capture_query_shapes = false` sends fingerprints only.

data/README.md

@@ -19,7 +19,6 @@ and [release notes](CHANGELOG.md).
 bundle install
 bin/rails generate rails_mind:install --plan
 bin/rails generate rails_mind:install
-bin/rails rails_mind:doctor
 ```
 
 Set **`RAILS_MIND_KEY`** for the destination application environment. It is the only
@@ -29,7 +28,14 @@ app's deployed `REVISION` file; telemetry still works when none is available.
 `RAILS_MIND_ENDPOINT` and `RAILS_MIND_RELEASE` are optional overrides. See
 [configuration and release detection](docs/sdk.md#configuration-and-release-detection).
 
-After setting the key, run `bin/rails rails_mind:verify` on the app's
+With the key in the app's environment, check the effective settings. Until the
+key is set, doctor names it and exits non-zero.
+
+```sh
+bin/rails rails_mind:doctor
+```
+
+Next, run `bin/rails rails_mind:verify` on the app's
 server. Match its check ID in RailsMind **Setup** and wait for **Check processed**.
 Then visit the app to confirm real traffic. This command sends one labeled
 diagnostic check, consumes one event of quota, and leaves business/performance

data/docs/sdk.md

@@ -16,6 +16,13 @@ gem "rails_mind", "~> 0.3.0"
 bundle install
 bin/rails generate rails_mind:install --plan
 bin/rails generate rails_mind:install
+```
+
+Then set `RAILS_MIND_KEY` ([configuration](#configuration-and-release-detection)) and, with
+it in the app's environment, check the effective settings. Until the key is set,
+doctor names it and exits non-zero.
+
+```sh
 bin/rails rails_mind:doctor
 ```
 
@@ -66,7 +73,9 @@ When upgrading an existing generated initializer, remove the old endpoint/releas
 assignments that directly read `ENV` so they do not overwrite the new defaults
 with `nil` or bypass platform detection. The installer preserves existing files.
 The doctor reports whether the effective endpoint, key, and release are present,
-without printing credential values or contacting the server.
+without printing credential values or contacting the server. While the key is
+missing or the endpoint is blank, it names the setting after the report and
+exits non-zero, as `rails_mind:verify` does.
 
 ## Verify the connection
 

data/lib/rails_mind/diagnostic.rb

@@ -36,5 +36,17 @@ module RailsMind
           "No network check is performed and credential values are never printed." ]
       }
     end
+
+    # What keeps collection off, one sentence each, naming the setting to change.
+    def problems
+      [].tap do |found|
+        if @config.token.to_s.strip.empty?
+          found << "RAILS_MIND_KEY is not set, so collection is off. Set it in this app's environment to the ingest key from RailsMind Setup, restart, then run bin/rails rails_mind:verify."
+        end
+        if @config.endpoint.to_s.strip.empty?
+          found << "The RailsMind endpoint is blank, so collection is off. Leave config.endpoint and RAILS_MIND_ENDPOINT unset to use #{Configuration::DEFAULT_ENDPOINT}, or set your RailsMind instance's URL."
+        end
+      end
+    end
   end
 end

data/lib/rails_mind/integrations/rails.rb

@@ -113,17 +113,71 @@ module RailsMind
       # was cut), numbers and hex literals become ?, and comments go. Double
       # quotes are identifiers, except on MySQL, where they can be strings.
       LEXEME = /(?<!\w)[Ee]'(?:[^'\\]|\\.|'')*'?|'(?:[^'\\]|\\.|'')*'?|\$(\w*)\$.*?(?:\$\1\$|\z)|"(?:[^"\\]|\\.|"")*"?|`[^`]*`?|--[^\n]*|\/\*.*?(?:\*\/|\z)|\b0[xX]\h+\b|\b\d+(?:\.\d+)?(?:[eE][-+]?\d+)?\b/m
+      # The same lexemes where a backslash escapes nothing outside E'...', as on
+      # PostgreSQL (Rails turns standard_conforming_strings on) and SQLite. Which
+      # rule applies is not always known (MySQL can turn backslash escapes off),
+      # so a statement with a backslash is read both ways and anything either
+      # reading takes for a literal goes. Without a backslash both agree.
+      STANDARD_LEXEME = /(?<!\w)[Ee]'(?:[^'\\]|\\.|'')*'?|'(?:[^']|'')*'?|\$(\w*)\$.*?(?:\$\1\$|\z)|"(?:[^"]|"")*"?|`[^`]*`?|--[^\n]*|\/\*.*?(?:\*\/|\z)|\b0[xX]\h+\b|\b\d+(?:\.\d+)?(?:[eE][-+]?\d+)?\b/m
 
       def self.normalize(sql, mysql: false)
-        sql.scrub("?").gsub(LEXEME) do |token|
-          case token[0]
-          when "-", "/" then " "
-          when "`" then token
-          when '"' then mysql ? "?" : token
-          else "?"
+        sql = sql.scrub("?")
+        text = sql.include?("\\") ? read_both_ways(sql, mysql) : sql.gsub(LEXEME) { |token| replacement(token, mysql) }
+        text.gsub(/\s+/, " ").strip
+      end
+
+      def self.replacement(token, mysql)
+        case token[0]
+        when "-", "/" then " "
+        when "`" then token
+        when '"' then mysql ? "?" : token
+        else "?"
+        end
+      end
+
+      # Literal spans from either reading merge into one ?; a comment that
+      # overlaps one joins it. Kept identifiers stay only where no reading saw a
+      # literal.
+      def self.read_both_ways(sql, mysql)
+        literals, comments = [], []
+        [ STANDARD_LEXEME, LEXEME ].each do |pattern|
+          sql.scan(pattern) do
+            token = Regexp.last_match
+            case replacement(token[0], mysql)
+            when "?" then literals << token.offset(0)
+            when " " then comments << token.offset(0)
+            end
+          end
+        end
+        literals = merge(literals)
+        loop do
+          # Merged literal spans are sorted and disjoint, so their ends are too.
+          joining, comments = comments.partition do |start, finish|
+            overlap = literals.bsearch { |_, last| last > start }
+            overlap && overlap[0] < finish
           end
-        end.gsub(/\s+/, " ").strip
+          break if joining.empty?
+          literals = merge(literals + joining)
+        end
+        spans = (literals.map { |span| span + [ "?" ] } + merge(comments).map { |span| span + [ " " ] }).sort
+        output, position = +"", 0
+        spans.each do |start, finish, mark|
+          output << sql[position...start] << mark
+          position = finish
+        end
+        output << sql[position..]
+      end
+
+      def self.merge(spans)
+        spans.sort.each_with_object([]) do |(start, finish), merged|
+          if merged.any? && start < merged.last[1]
+            merged.last[1] = [ merged.last[1], finish ].max
+          else
+            merged << [ start, finish ]
+          end
+        end
       end
+      private_class_method :replacement, :read_both_ways, :merge
 
       class << self
         def install!

data/lib/rails_mind/redactor.rb

@@ -2,13 +2,22 @@ module RailsMind
   class Redactor
     FILTER = /password|passwd|secret|token|api[_-]?key|access[_-]?key|private[_-]?key|authorization|cookie|credential|session|email|phone|address|credit|card|body|params|arguments|bind|sql|db\.statement|query\.text/i
     MAX_STRING = 1024
+    # Scrubbing reads at most this much of a string; what it sends is still at
+    # most MAX_STRING. Every pass below is linear, and the cap bounds the rest.
+    INPUT_LIMIT = 8192
+    # Identity, correlation and release values are not free text: a numeric
+    # account ID that passes the Luhn check is not a card number.
+    PLAIN_FIELDS = %w[id kind occurred_at status release trace_id span_id parent_span_id request_id job_id visitor_id user_id account_id].freeze
 
     def call(value, depth = 0)
       return "[TRUNCATED]" if depth > 5
       case value
       when Hash
         value.first(50).to_h.transform_keys { |key| key.to_s[0, 100] }.each_with_object({}) do |(key, item), clean|
-          clean[key] = key.match?(FILTER) ? "[FILTERED]" : call(item, depth + 1)
+          clean[key] = if key.match?(FILTER) then "[FILTERED]"
+          elsif depth.zero? && PLAIN_FIELDS.include?(key) && (item.is_a?(String) || item.is_a?(Symbol)) then scrub(item.to_s, free_text: false)
+          else call(item, depth + 1)
+          end
         end
       when Array then value.first(50).map { |item| call(item, depth + 1) }
       when String then scrub(value)
@@ -19,30 +28,55 @@ module RailsMind
       end
     end
 
+    # An authorization scheme names the credential that follows it, so the
+    # scheme and the credential go together. Digest and AWS signature headers
+    # carry several credential parts, so their whole value goes.
+    SCHEME = "(?:bearer|basic|token|negotiate|ntlm|bot|apikey)\\s+".freeze
+    WHOLE_VALUE_SCHEME = "(?:digest|aws4-hmac-sha256)\\s[^\\n]*".freeze
+
     # Exception messages carry what generic scrubbing cannot recognize: database
     # errors quote the row, key or input that failed (and the SQL with its
-    # literals), and Ruby prints hashes and keyword arguments with their values.
+    # literals), and Ruby prints hashes and keyword arguments with their values,
+    # header values wrapped in arrays (Net::HTTP#to_hash) included.
     DATABASE_ERROR = /\A(?:PG|Mysql2|Trilogy|SQLite3)::|StatementInvalid|RecordNotUnique|NotNullViolation|ForeignKeyViolation|ValueTooLong|RangeError\z/
-    SENSITIVE_VALUE = /((?:"|:)?(?:password|passwd|secret|token|api[_-]?key|access[_-]?key|private[_-]?key|authorization|credential|cookie|session)\w*"?\s*(?:=>|:)\s*)(?:"(?:[^"\\]|\\.)*"|'(?:[^'\\]|\\.)*'|[^\s,}\])]+)/i
+    SENSITIVE_VALUE = /((?:"|:)?(?:password|passwd|secret|token|api[_-]?key|access[_-]?key|private[_-]?key|authorization|credential|cookie|session|phone|ssn)\w{0,64}"?\s*(?:=>|:)\s*(?:\[(?!FILTERED\]))?)(?:\[FILTERED\]|"(?:[^"\\]|\\.)*"|'(?:[^'\\]|\\.)*'|#{WHOLE_VALUE_SCHEME}|(?:#{SCHEME})?[^\s,}\])]+)/i
+
+    # PostgreSQL connections use standard_conforming_strings, where a backslash
+    # does not escape the quote of a '...' literal (it does inside E'...');
+    # MySQL lets it escape the quote. Text with a backslash is read both ways,
+    # and anything either reading takes for a literal goes.
+    STANDARD_LITERAL = /(?<!\w)[Ee]'(?:[^'\\]|\\.|'')*'?|'(?:[^']|'')*'?/
+    ESCAPED_LITERAL = /'(?:[^'\\]|\\.|'')*'?/
 
     def scrub_message(error)
-      text = error.message.to_s.encode("UTF-8", invalid: :replace, undef: :replace, replace: "�")
+      text = bounded(error.message.to_s.encode("UTF-8", invalid: :replace, undef: :replace, replace: "�"))
       text = text.gsub(SENSITIVE_VALUE, '\\1"[FILTERED]"')
         .gsub(/(Failing row contains )\([^\n]*\)/, "\\1(…)")
         .gsub(/(Key \([^)\n]*\)=)\([^\n]*\)/, "\\1(…)")
         .gsub(/(invalid input (?:syntax|value) for [^:\n]*: )"[^"\n]*"/, '\\1"…"')
-      text = text.gsub(/'(?:[^'\\]|\\.|'')*'?/, "?") if database_error?(error)
+      text = mask_literals(text) if database_error?(error)
       scrub(text)
     end
 
-    def scrub(value)
-      value.encode("UTF-8", invalid: :replace, undef: :replace, replace: "�")
-        .gsub(/\b[A-Z0-9._%+\-]+@[A-Z0-9.\-]+\.[A-Z]{2,}\b/i, "[EMAIL]")
+    # Userinfo runs to the last @ before the path, so an unencoded @ in a
+    # password cannot leave its tail behind.
+    USERINFO = %r{\b([a-z][a-z0-9+.\-]{0,31}://)[^\s/]*@}i
+    EMAIL = /\b[A-Z0-9._%+\-]{1,128}@[A-Z0-9.\-]{1,253}\.[A-Z]{2,}\b/i
+    AUTHORIZATION = /((?:proxy-)?authorization["']?\s*(?:=>|[=:])\s*(?:\[(?!FILTERED\]))?\s*["']?)(?:\[FILTERED\]|#{WHOLE_VALUE_SCHEME}|(?:#{SCHEME})?[^\s,;&"']+)/i
+    KEY_VALUE = /((?:password|secret|token|api[_-]?key|access[_-]?key|private[_-]?key|authorization)\s*(?:=>|[=:])\s*)(?:#{SCHEME})?[^\s,;&]+/i
+    PROVIDER_TOKEN = /\b(?:gh[pousr]_[A-Za-z0-9]{36,}|github_pat_\w{22,}|(?:AKIA|ASIA)[A-Z0-9]{16}\b|[sr]k_(?:live|test)_[A-Za-z0-9]{10,}|xox[abposr]-[A-Za-z0-9-]{10,}|sk-(?:proj-|ant-)?[A-Za-z0-9_-]{20,}|eyJ[\w-]{5,512}\.[\w-]{5,}\.[\w-]*)/
+    SSN = /\b\d{3}([- ])\d{2}\1\d{4}\b/
+    DIGIT_RUN = /(?<!\w)\d(?:[ -]?\d)*(?!\w)/
+
+    def scrub(value, free_text: true)
+      text = bounded(value.encode("UTF-8", invalid: :replace, undef: :replace, replace: "�"))
+        .gsub(USERINFO, '\\1[FILTERED]@')
+        .gsub(EMAIL, "[EMAIL]")
+        .gsub(AUTHORIZATION, '\\1[FILTERED]')
         .gsub(/\bBearer\s+\S+/i, "Bearer [FILTERED]")
-        .gsub(/((?:password|secret|token|api[_-]?key|access[_-]?key|private[_-]?key|authorization)\s*[=:]\s*)[^\s,;&]+/i, '\\1[FILTERED]')
-        .gsub(%r{(https?://)[^\s/@]+:[^\s/@]+@}, '\\1[FILTERED]@')
-        .gsub(%r{(https?://[^\s?#]+)[?#][^\s]*}, '\\1')
-        .slice(0, MAX_STRING)
+        .gsub(KEY_VALUE, '\\1[FILTERED]')
+      text = text.gsub(PROVIDER_TOKEN, "[FILTERED]").gsub(SSN, "[FILTERED]").gsub(DIGIT_RUN) { |run| mask_cards(run) } if free_text
+      text.gsub(%r{https?://\S+}) { |url| url[/\A[^?#]*/] }.slice(0, MAX_STRING)
     end
 
     private
@@ -50,5 +84,85 @@ module RailsMind
     def database_error?(error)
       error.class.ancestors.any? { |ancestor| ancestor.name.to_s.match?(DATABASE_ERROR) }
     end
+
+    # A cut string ends at its last whitespace, so no half token is left that a
+    # pass could no longer recognize.
+    def bounded(text)
+      return text if text.length <= INPUT_LIMIT
+      text = text[0, INPUT_LIMIT]
+      cut = text.rindex(/\s/)
+      cut ? text[0, cut] : text
+    end
+
+    def mask_literals(text)
+      return text.gsub(ESCAPED_LITERAL, "?") unless text.include?("\\")
+      spans = [ STANDARD_LITERAL, ESCAPED_LITERAL ].flat_map { |pattern| text.to_enum(:scan, pattern).map { Regexp.last_match.offset(0) } }
+      output, position = +"", 0
+      merge(spans).each do |start, finish|
+        output << text[position...start] << "?"
+        position = finish
+      end
+      output << text[position..]
+    end
+
+    def merge(spans)
+      spans.sort.each_with_object([]) do |(start, finish), merged|
+        if merged.any? && start < merged.last[1]
+          merged.last[1] = [ merged.last[1], finish ].max
+        else
+          merged << [ start, finish ]
+        end
+      end
+    end
+
+    # A card number is 13 to 19 digits, maybe in groups split by spaces or
+    # dashes, that pass the Luhn check. A run of digit groups can hold one next
+    # to other numbers (an expiry, a quantity), so every span of whole groups
+    # with at most 19 digits is tried, and the longest that passes goes.
+    def mask_cards(run)
+      groups = run.to_enum(:scan, /\d+/).map { Regexp.last_match.offset(0) }
+      digits = run.delete(" -")
+      return run if digits.length < 13
+      bounds, count = [], 0
+      groups.each { |start, finish| bounds << [ count, count += finish - start ] }
+      plain, doubled = luhn_sums(digits)
+      output, position, first = +"", 0, 0
+      while first < groups.size
+        longest, from = nil, bounds[first][0]
+        (first...groups.size).each do |last|
+          to = bounds[last][1]
+          break if to - from > 19
+          next if to - from < 13
+          parity = to % 2
+          sum = doubled[parity][to] - doubled[parity][from] + plain[1 - parity][to] - plain[1 - parity][from]
+          longest = last if (sum % 10).zero?
+        end
+        if longest
+          output << run[position...groups[first][0]] << "[FILTERED]"
+          position, first = groups[longest][1], longest + 1
+        else
+          first += 1
+        end
+      end
+      output << run[position..]
+    end
+
+    # Prefix sums of each digit, plain and Luhn-doubled, split by position
+    # parity. In a span ending at index `to`, Luhn doubles the digits whose
+    # position has the parity of `to`, so any span's checksum is four lookups.
+    def luhn_sums(digits)
+      plain, doubled = Array.new(2) { [ 0 ] }, Array.new(2) { [ 0 ] }
+      digits.each_char.with_index do |char, index|
+        value = char.ord - 48
+        twice = value * 2
+        twice -= 9 if twice > 9
+        [ 0, 1 ].each do |parity|
+          own = index % 2 == parity
+          plain[parity] << plain[parity].last + (own ? value : 0)
+          doubled[parity] << doubled[parity].last + (own ? twice : 0)
+        end
+      end
+      [ plain, doubled ]
+    end
   end
 end

data/lib/rails_mind/version.rb

@@ -1,3 +1,3 @@
 module RailsMind
-  VERSION = "0.3.1"
+  VERSION = "0.3.2"
 end

data/lib/tasks/rails_mind.rake

@@ -7,9 +7,15 @@ namespace :rails_mind do
     exit 1 unless result[:success]
   end
 
-  desc "Inspect dependencies and report the non-destructive integration plan"
+  desc "Inspect dependencies and report the non-destructive integration plan; fails while collection is off"
   task doctor: :environment do
     require "rails_mind/diagnostic"
-    puts JSON.pretty_generate(RailsMind::Diagnostic.new(root: Rails.root).report.merge(collector: RailsMind.stats))
+    diagnostic = RailsMind::Diagnostic.new(root: Rails.root)
+    puts JSON.pretty_generate(diagnostic.report.merge(collector: RailsMind.stats))
+    problems = diagnostic.problems
+    unless problems.empty?
+      warn problems
+      exit 1
+    end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rails_mind
 version: !ruby/object:Gem::Version
-  version: 0.3.1
+  version: 0.3.2
 platform: ruby
 authors:
 - RailsMind