squelch 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 3882fdd525cffcfe62b3ac46dac3af8ebce60ea3c83e69315be51ac211dce232
+  data.tar.gz: 7e82cc70e6341afd6de48aa8d67c55d4ce618d06f62e52aff7c4fdd6f1c202cc
+SHA512:
+  metadata.gz: 6bd3863b88e5ed602d2f632c4f67c6f339032d743d47281b8a0041dfb2e0ff47b335426c8dfbb88cc15e1ff8d7b2c82b37c7d0e58e102e597a6cc4a0a08021b0
+  data.tar.gz: 3c48c016f881d621db4ddad51883e29b5cf5ac8aa2db58baaf225904d1a46633e26489b8f6de6c274556f401d931082296911f423f3e62c2f74040e95a209fda

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2021 Alex Vondrak
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,88 @@
+# Squelch
+
+[![build](https://github.com/ajvondrak/squelch/workflows/build/badge.svg)](https://github.com/ajvondrak/squelch/actions?query=workflow%3Abuild)
+[![coverage](https://coveralls.io/repos/github/ajvondrak/squelch/badge.svg?branch=main)](https://coveralls.io/github/ajvondrak/squelch?branch=main)
+
+Squelch squelches SQL!
+
+```sql
+-- Before
+INSERT INTO users(name, address, phone) VALUES ("John Doe", "1600 Pennsylvania Ave", "867-5309");
+
+-- After
+INSERT INTO users(name, address, phone) VALUES (?, ?, ?);
+```
+
+This gem is a purposefully simple string obfuscator. It aims to replace every data literal in a SQL query with a `?` placeholder, as though it were a prepared statement. The result should still be readable SQL, but without the risk of leaking potentially sensitive information.
+
+The code was originally adapted from the [`NewRelic::Agent::Database::ObfuscationHelpers`](https://github.com/newrelic/newrelic-ruby-agent/blob/f0290ab6468ad205dd014d63c794883dc47eebe7/lib/new_relic/agent/database/obfuscation_helpers.rb) in the [newrelic\_rpm](https://rubygems.org/gems/newrelic_rpm) gem. By abstracting out these low-level implementation details, the hope is that Squelch can empower other libraries to easily sanitize their SQL logs.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem "squelch"
+```
+
+and then install it with `bundle install`.
+
+Alternatively, you could install it to your system's gems with:
+
+```console
+$ gem install squelch
+```
+
+## Usage
+
+### Basic interface
+
+The main API is the `Squelch.obfuscate` method, which takes in your SQL string and returns an obfuscated version of it.
+
+```ruby
+Squelch.obfuscate("SELECT * FROM social_security_cards WHERE number = 'pii';")
+
+#=> "SELECT * FROM social_security_cards WHERE number = ?;"
+```
+
+This method is powered by regular expression patterns, some of which correspond to particular database systems. For example, Postgres supports a unique [dollar quoting](https://www.postgresql.org/docs/13/sql-syntax-lexical.html#SQL-SYNTAX-DOLLAR-QUOTING) syntax, while Oracle has its own [Q quoting](https://livesql.oracle.com/apex/livesql/file/content_CIREYU9EA54EOKQ7LAMZKRF6P.html) syntax. If possible, try to always supply the optional `db:` keyword parameter with a symbol corresponding to your RDMS. The currently supported options are `:mysql`, `:postgres`, `:sqlite`, `:oracle`, and `:cassandra`, but any other option will fall back safely to a generic default pattern.
+
+```ruby
+Squelch.obfuscate("SELECT * FROM credit_cards WHERE number = $pii$ ... $pii$;", db: :postgres)
+
+#=> "SELECT * FROM credit_cards WHERE number = ?;"
+```
+
+```ruby
+Squelch.obfuscate("SELECT * FROM phones WHERE number = q'<pii>';", db: :oracle)
+
+#=> "SELECT * FROM phones WHERE number = ?;"
+```
+
+### Handling errors
+
+When there's an issue with squelching the SQL, we don't want to risk of using the problematic results that might still be leaking PII. The error-safe `Squelch.obfuscate` method returns a single `?` placeholder in the event of an issue, but Squelch has the error-raising variant `Squelch.obfuscate!` as well.
+
+```ruby
+Squelch.obfuscate("SELECT * FROM table WHERE pii = 'a string missing a closing quote;")
+
+#=> "?"
+```
+
+```ruby
+Squelch.obfuscate!("SELECT * FROM table WHERE pii = 'a string missing a closing quote;")
+
+#=> Squelch::Error: Failed to squelch SQL, delimiter ' remained after obfuscation
+```
+
+If you rescue the `Squelch::Error`, you can access the problematic obfuscation result in `Squelch::Error#obfuscation`.
+
+```ruby
+begin
+  Squelch.obfuscate!("SELECT * FROM users WHERE id = 12345 AND name = 'Mister Danglin' Quote';")
+rescue Squelch::Error => e
+  e.obfuscation
+end
+
+#=> "SELECT * FROM users WHERE id = ? AND name = ? Quote';"
+```

data/lib/squelch.rb

@@ -0,0 +1,109 @@
+# frozen_string_literal: true
+
+require "squelch/version"
+require "squelch/pairs"
+require "squelch/patterns"
+require "squelch/database"
+require "squelch/error"
+
+# A simple SQL obfuscator.
+#
+# The goal of {Squelch} is to replace every data literal in any given SQL
+# string with a placeholder `?`. This removes any potentially sensitive values
+# by removing *all* values, making the SQL read like it was a prepared
+# statement all along.
+#
+# However, this process might produce bad results if there's a bug or you give
+# a malformed SQL query. One heuristic we use to try to catch these cases is to
+# look at the end result: are there any unmatched delimiters left over? Since
+# obfuscation should remove *all* strings and comments, any dangling quotes or
+# `/*` markers or such would indicate that either:
+#
+# * the SQL query had mismatched delimiter pairs in the first place, or
+# * the obfuscation patterns failed to parse all the data literals correctly.
+#
+# Either way, we should be cautious of using either the original or the
+# improperly-obfuscated query, since both may still contain sensitive
+# information. To deal with these issues, we have both the error-safe
+# {Squelch.obfuscate} and the error-raising {Squelch.obfuscate!}.
+#
+# Both methods not only accept a string of SQL, but optionally a specific
+# database driver. This is because certain databases have their own special
+# syntax. For example, Postgres uses double quotes around table names and
+# supports `$$dollar quoted$$` strings, whereas MySQL uses backticks around
+# table names and supports `"double quoted"` strings.
+#
+# To stand the best chance of scrubbing all sensitive values from your SQL, you
+# should provide the specific database that you're using. That way, {Squelch}
+# can make tweaks to its internal pattern matching. The default just tries to
+# match all possible special cases, but that may wind up obfuscating too much
+# or even being less performant than a more specific option.
+#
+# @example Basic obfuscation
+#   Squelch.obfuscate("SELECT * FROM examples WHERE name = 'basic';")
+#   #=> "SELECT * FROM examples WHERE name = ?;"
+#
+# @example Malformed query
+#   Squelch.obfuscate("SELECT * FROM examples WHERE name = ''malformed';")
+#   #=> "?"
+#
+#   begin
+#     Squelch.obfuscate!("SELECT * FROM examples WHERE name = ''malformed';")
+#   rescue Squelch::Error => e
+#     puts e.message
+#     puts
+#     puts e.obfuscation
+#   end
+#   # Failed to squelch SQL, delimiter ' remained after obfuscation
+#   #
+#   # SELECT * FROM examples WHERE name = ?malformed';
+#
+# @example Using database-specific syntax
+#   Squelch.obfuscate(
+#     'SELECT "examples".name FROM examples WHERE db = $$postgres$$;',
+#     db: :mysql,
+#   )
+#   #=> "SELECT ?.name FROM examples WHERE db = $$postgres$$;"
+#
+#   Squelch.obfuscate(
+#     'SELECT "examples".name FROM examples WHERE db = $$postgres$$;',
+#     db: :postgres,
+#   )
+#   #=> 'SELECT "examples".name FROM examples WHERE db = ?;'
+module Squelch
+  # Obfuscates a SQL query.
+  #
+  # If the resulting obfuscation still has dangling delimiters left over, we
+  # return a single placeholder for the whole query, `?`. In order to get more
+  # information about such errors, you should use {.obfuscate!}.
+  #
+  # @param sql [String] an unobfuscated SQL query
+  #
+  # @param db [Symbol] the specific database syntax being used; supports
+  #   `:mysql`, `:postgres`, `:sqlite`, `:oracle`, `:oracle`, or `:cassandra`,
+  #   while anything else will be treated as `:default`
+  #
+  # @return [String] the obfuscated SQL query
+  def self.obfuscate(sql, db: :default)
+    obfuscate!(sql, db: db)
+  rescue Error
+    "?"
+  end
+
+  # Obfuscates a SQL query, raising an error if there are issues.
+  #
+  # If the resulting obfuscation still has dangling delimiters left over, we
+  # raise {Squelch::Error} with more information about what went wrong. If you
+  # don't care about the details and just want some canonical string output,
+  # you should use {.obfuscate}.
+  #
+  # @param (see .obfuscate)
+  # @return (see .obfuscate)
+  # @raise [Squelch::Error] if obfuscation didn't remove all delimiters
+  def self.obfuscate!(sql, db: :default)
+    sql.gsub(Database.pattern(db), "?").tap do |obfuscated|
+      mismatched = obfuscated.match(Database.pairs(db))
+      raise Error.new(sql, mismatched) if mismatched
+    end
+  end
+end

data/lib/squelch/database.rb

@@ -0,0 +1,101 @@
+# frozen_string_literal: true
+
+module Squelch
+  # @private
+  module Database
+    PATTERNS = {
+      mysql: Regexp.union(
+        Patterns::SINGLE_QUOTED,
+        Patterns::DOUBLE_QUOTED,
+        Patterns::NUMBER,
+        Patterns::BOOLEAN,
+        Patterns::HEXADECIMAL,
+        Patterns::LINE_COMMENT,
+        Patterns::BLOCK_COMMENT,
+      ).freeze,
+      postgres: Regexp.union(
+        Patterns::SINGLE_QUOTED,
+        Patterns::DOLLAR_QUOTED,
+        Patterns::UUID,
+        Patterns::NUMBER,
+        Patterns::BOOLEAN,
+        Patterns::LINE_COMMENT,
+        Patterns::BLOCK_COMMENT,
+      ).freeze,
+      sqlite: Regexp.union(
+        Patterns::SINGLE_QUOTED,
+        Patterns::NUMBER,
+        Patterns::BOOLEAN,
+        Patterns::HEXADECIMAL,
+        Patterns::LINE_COMMENT,
+        Patterns::BLOCK_COMMENT,
+      ).freeze,
+      oracle: Regexp.union(
+        Patterns::SINGLE_QUOTED,
+        Patterns::ORACLE_QUOTED,
+        Patterns::NUMBER,
+        Patterns::LINE_COMMENT,
+        Patterns::BLOCK_COMMENT,
+      ).freeze,
+      cassandra: Regexp.union(
+        Patterns::SINGLE_QUOTED,
+        Patterns::UUID,
+        Patterns::NUMBER,
+        Patterns::BOOLEAN,
+        Patterns::HEXADECIMAL,
+        Patterns::LINE_COMMENT,
+        Patterns::BLOCK_COMMENT,
+      ).freeze,
+      default: Regexp.union(
+        Patterns::SINGLE_QUOTED,
+        Patterns::DOUBLE_QUOTED,
+        Patterns::DOLLAR_QUOTED,
+        Patterns::UUID,
+        Patterns::NUMBER,
+        Patterns::BOOLEAN,
+        Patterns::HEXADECIMAL,
+        Patterns::LINE_COMMENT,
+        Patterns::BLOCK_COMMENT,
+        Patterns::ORACLE_QUOTED,
+      ).freeze,
+    }.freeze
+
+    def self.pattern(db)
+      PATTERNS[db] || PATTERNS[:default]
+    end
+
+    PAIRS = {
+      mysql: Regexp.union(
+        Pairs::SINGLE_QUOTED,
+        Pairs::DOUBLE_QUOTED,
+        Pairs::BLOCK_COMMENT,
+      ),
+      postgres: Regexp.union(
+        Pairs::SINGLE_QUOTED,
+        Pairs::DOLLAR_QUOTED,
+        Pairs::BLOCK_COMMENT,
+      ),
+      sqlite: Regexp.union(
+        Pairs::SINGLE_QUOTED,
+        Pairs::BLOCK_COMMENT,
+      ),
+      cassandra: Regexp.union(
+        Pairs::SINGLE_QUOTED,
+        Pairs::BLOCK_COMMENT,
+      ),
+      oracle: Regexp.union(
+        Pairs::SINGLE_QUOTED,
+        Pairs::BLOCK_COMMENT,
+      ),
+      default: Regexp.union(
+        Pairs::SINGLE_QUOTED,
+        Pairs::DOUBLE_QUOTED,
+        Pairs::BLOCK_COMMENT,
+      ),
+    }.freeze
+
+    def self.pairs(db)
+      PAIRS[db] || PAIRS[:default]
+    end
+  end
+end

data/lib/squelch/error.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+module Squelch
+  # Raised by {Squelch.obfuscate!} if obfuscation seems to have failed.
+  #
+  # This might be raised either because the SQL was malformed in the first
+  # place or because of a bug in our parsing. See {Squelch} for more
+  # discussion.
+  class Error < StandardError
+    # @return [String] the original SQL input
+    attr_reader :sql
+
+    # @return [String] the invalid result of obfuscating {#sql}
+    attr_reader :obfuscation
+
+    # @return [String] the left over delimiter detected in {#obfuscation}
+    attr_reader :delimiter
+
+    def initialize(sql, mismatched)
+      @sql = sql
+      @obfuscation = mismatched.string
+      @delimiter = mismatched.to_s
+      super(<<~MSG.strip)
+        Failed to squelch SQL, delimiter #{delimiter} remained after obfuscation
+      MSG
+    end
+  end
+end

data/lib/squelch/pairs.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+module Squelch
+  # @private
+  module Pairs
+    SINGLE_QUOTED = "'"
+    DOUBLE_QUOTED = '"'
+    DOLLAR_QUOTED = /\$(?!\?)/.freeze
+    BLOCK_COMMENT = Regexp.union("/*", "*/").freeze
+  end
+end

data/lib/squelch/patterns.rb

@@ -0,0 +1,81 @@
+# frozen_string_literal: true
+
+module Squelch
+  # @private
+  module Patterns
+    ORACLE_QUOTED = Regexp.union(
+      /q'\<.*?\>'/m, # q'<text>'
+      /q'\[.*?\]'/m, # q'[text]'
+      /q'\{.*?\}'/m, # q'{text}'
+      /q'\(.*?\)'/m, # q'(text)'
+    ).freeze
+
+    SINGLE_QUOTED = %r{
+      '             # a single quote
+      (?:           # followed by zero or more
+        [^']        #   non-quote characters
+        |           #   or
+        ''          #   escaped quotes
+      )*?           #
+      (?:           # and closed by either
+        \\'.*       #   a literal backslash at the end of the string
+        |           #   or
+        '(?!')      #   one non-escaped quote
+      )             #
+    }mx.freeze
+
+    DOUBLE_QUOTED = %r{
+      "             # a double quote
+      (?:           # followed by zero or more
+        [^"]        #   non-quote characters
+        |           #   or
+        ""          #   escaped quotes
+      )*?           #
+      (?:           # and closed by either
+        \\".*       #   a literal backslash at the end of the string
+        |           #   or
+        "(?!")      #   one non-escaped quote
+      )             #
+    }mx.freeze
+
+    DOLLAR_QUOTED = %r{
+      (             # a dollar quote is defined dynamically
+        \$          # the quote mark begins with a dollar sign
+        (?!\d)      #   unless followed by a digit, which denotes a variable
+        [^$]*?      #   otherwise it can can optionally contain any characters
+        \$          # up until another dollar sign
+      )             #
+      .*?           # there's some amount of intervening text being quoted
+      \1            # until the dollar quote mark is repeated
+    }mx.freeze
+
+    LINE_COMMENT = %r{
+      (?:\#|--)     # single-line comments start with a hash or double hyphen
+      .*?           # they contain arbitrary text
+      (?=\r|\n|$)   # up until, but not including, the newline/carriage-return
+    }x.freeze
+
+    BLOCK_COMMENT = %r{
+      /\*           # block comments open with a slash-star
+        (?>         #   they contain zero or more
+          [^/*]     #   non-star or -slash characters
+          |         #   or
+          /(?!\*)   #   a slash as long as it's not followed by a star
+          |         #   or
+          \*(?!/)   #   a star as long as it's not followed by a slash
+          |         #   or
+          \g<0>     #   recursively, a nested block comment
+        )*          #
+        /?          #   slash followed by star is ok if star is part of close
+      \*/           # block comments close with a star-slash
+    }mx.freeze
+
+    UUID = /\{?(?:[0-9a-fA-F]-*){32}\}?/.freeze
+
+    NUMBER = /-?\b(?:[0-9]+\.)?[0-9]+([eE][+-]?[0-9]+)?\b/.freeze
+
+    BOOLEAN = /\b(?:true|false|null)\b/i.freeze
+
+    HEXADECIMAL = /0x\h+/.freeze
+  end
+end

data/lib/squelch/version.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+module Squelch
+  # The current version of the gem.
+  VERSION = "0.1.0"
+end

metadata

@@ -0,0 +1,60 @@
+--- !ruby/object:Gem::Specification
+name: squelch
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Alex Vondrak
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2021-03-05 00:00:00.000000000 Z
+dependencies: []
+description: |
+  Squelch squelches SQL!
+
+  This gem is a purposefully simple string obfuscator. It aims to replace
+  every data literal in a SQL query with a `?` placeholder, as though it were
+  a prepared statement. The result should still be readable SQL, but without
+  the risk of leaking potentially sensitive information.
+email: ajvondrak@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- LICENSE
+- README.md
+- lib/squelch.rb
+- lib/squelch/database.rb
+- lib/squelch/error.rb
+- lib/squelch/pairs.rb
+- lib/squelch/patterns.rb
+- lib/squelch/version.rb
+homepage: https://github.com/ajvondrak/squelch
+licenses:
+- MIT
+metadata:
+  bug_tracker_uri: https://github.com/ajvondrak/squelch/issues
+  documentation_uri: https://rubydoc.info/github/ajvondrak/squelch/main
+  homepage_uri: https://github.com/ajvondrak/squelch
+  source_code_uri: https://github.com/ajvondrak/squelch
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.6.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.2.3
+signing_key:
+specification_version: 4
+summary: A simple SQL obfuscator
+test_files: []