tty-logger 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 31142537abe01781e199d8b551eea32bdc802c1e9a46bec7947b3240249df798
-  data.tar.gz: 33b6816da36b2c41651452a9efa82b8e69b62eef6dbdfdc136c5195140b76d4b
+  metadata.gz: af3524110f6873609fc2086063371a728bd63d08d9f9160c8469cef4e5e50d94
+  data.tar.gz: 4991ed5e387b34fbe8c2e1887ba36ce53e5bc3bc120f957eda6a778202abcf35
 SHA512:
-  metadata.gz: d5b65cbbe0a3c9cefcb3fe082242a47e6090626c41531fbd6f97dfc528e795b2e2f56f2ce5f27f1a0404620ce2a12139745e028a9d3a31a6bf630bb7c600f3fb
-  data.tar.gz: 9e19ee528817ea3ef0d4ccf5c4a9601857b69e3b2120ad8ca7b5715ca57ea164a346ecd7cd90ad253cc81b707820098a019b77ea54dff547e12e8384b7154477
+  metadata.gz: 66c6892da31f674bcdee76a92334976f19f4447e341cf61c5fce106d5ad7085035d5a580c80f614bc564f4f6567ae6a446aacfe17d7499fb7c65fd4e800d0003
+  data.tar.gz: 45ee01d044d21a79a85fe55933ef3da2c3aefd58cea31453d61f7007768687e9256447e214de0f90973dc4de136f35eba735e9619c0c17aa9d74b05a32d84090

data/CHANGELOG.md

@@ -1,5 +1,16 @@
 # Change log
 
+## [v0.3.0] - 2020-01-01
+
+### Added
+* Add ability to filter sensitive information out of structured data
+
+### Changed
+* Remove the test and task files from the gemspec
+
+### Fixed
+* Fix console handler highlighting of nested hash keys
+
 ## [v0.2.0] - 2019-09-30
 
 ### Added
@@ -16,5 +27,6 @@
 
 * Initial implementation and release
 
+[v0.3.0]: https://github.com/piotrmurach/tty-logger/compare/v0.2.0..v0.3.0
 [v0.2.0]: https://github.com/piotrmurach/tty-logger/compare/v0.1.0..v0.2.0
 [v0.1.0]: https://github.com/piotrmurach/tty-logger/compare/v0.1.0

data/README.md

@@ -375,15 +375,17 @@ The `:metdata` configuration option can include the following symbols:
 
 ### 2.4.2 Filters
 
-You can filter sensitive data out of log output with `filters` configuration option. As a value provide a list of sensitive data items:
+You can filter sensitive data out of log output with `filters` configuration option. The `filters` can be further configured to remove info from log message with `message` or structured data with `data`. Both methods, as a value accept a list of sensitive items to search for.
+
+If you want to filter sensitive information from log messages use `message`:
 
 ```ruby
 logger = TTY::Logger.new(output: output) do |config|
-  config.filters = ["secret", "password"]
+  config.filters.message = %w[secret password]
 end
 ```
 
-Which by default will replace each data item with `[FILTERED]` placeholder:
+Which by default will replace each matching string with `[FILTERED]` placeholder:
 
 ```ruby
 logger.info("Super secret info with password")
@@ -391,13 +393,14 @@ logger.info("Super secret info with password")
 # ℹ info    Super [FILTERED] info with [FILTERED]
 ```
 
-You can also replace each data item with a custom placeholder. To do so use a hash with pairs of matched text and replacement placeholder.
+You can also replace each data item with a custom placeholder. To do so use a `:mask` keyword with a replacement placeholder.
 
-For example, to replace "secret" content with placeholder "<SECRET>" do:
+For example, to replace "secret" content with placeholder `"<SECRET>"` do:
 
 ```ruby
 logger = TTY::Logger.new do |config|
-  config.filters = { "secret" => "<SECRET>" }
+  config.filters.message = %w[secret]
+  config.filters.mask = "<SECRET>"
 end
 ```
 
@@ -409,6 +412,64 @@ logger.info("Super secret info")
 # ℹ info    Super <SECRET> info
 ```
 
+To filter out sensitive information out of structured data use `data` method. By default any value matching a parameter name will be filtered regardless of the level of nesting. If you wish to filter only a specific deeply nested key use a dot notation like `params.card.password` to only filter `{params: {card: {password: "Secret123"}}}`.
+
+For example to filter out a `:password` from data do:
+
+```ruby
+logger = TTY::Logger.new do |config|
+  config.filters.data = %i[password]
+end
+```
+
+This will filter out any key matching password:
+
+```ruby
+logger.info("Secret info", password: "Secret123", email: "")
+# =>
+# ℹ info    Secret info     password="[FILTERED]" email="secret@example.com"
+```
+
+But also any nested data item:
+
+```ruby
+logger.info("Secret info", params: {password: "Secret123", email: ""})
+# =>
+# ℹ info    Secret info     params={password="[FILTERED]" email="secret@example.com"}
+```
+
+You're not limited to using only direct string comparison. You can also match based on regular expressions. For example, to match keys starting with `ba` we can add a following filter:
+
+```ruby
+logger = TTY::Logger.new do |config|
+  config.filters.data = [/ba/]
+end
+```
+
+Then appropriate values will be masked:
+
+```ruby
+logger.info("Filtering data", {"foo" => {"bar" => "val", "baz" => "val"}})
+# =>
+# ℹ info    Filtering data            foo={bar="[FILTERED]" baz="[FILTERED]"}
+```
+
+You can mix and match. To filter keys based on pattern inside a deeply nested hash use dot notation with regular expression. For example, to find keys for the `:foo` parent key that starts with `:b` character, we could do:
+
+```ruby
+logger = TTY::Logger.new do |config|
+  config.filters.data = [/^foo\.b/]
+end
+```
+
+Then only keys under the `:foo` key will be filtered:
+
+```ruby
+logger.info("Filtering data", {"foo" => {"bar" => "val"}, "baz" => {"bar" => val"}})
+# =>
+# ℹ info    Filtering data            foo={bar="[FILTERED]"} baz={bar=val}
+```
+
 ### 2.5 Cloning
 
 You can create a copy of a logger with the current configuration using the `copy` method.
@@ -547,7 +608,7 @@ end
 By default, the output will be a plain text streamed to console. The text contains key and value pairs of all the metadata and the message of the log event.
 
 ```ruby
-loggger.info("Info about the deploy", app:"myap", env:"prod")
+logger.info("Info about the deploy", app:"myap", env:"prod")
 # =>
 # pid=18315 date="2019-07-21" time="15:42:12.463" path="examples/stream.rb:17:in`<main>`"
 # level=info message="Info about the deploy" app=myapp env=prod
@@ -565,7 +626,7 @@ end
 This will output JSON formatted text streamed to console.
 
 ```ruby
-loggger.info("Info about the deploy", app="myap", env="prod")
+logger.info("Info about the deploy", app="myap", env="prod")
 # =>
 # {"pid":18513,"date":"2019-07-21","time":"15:54:09.924","path":"examples/stream.rb:17:in`<main>`",
 # "level":"info","message":"Info about the deploy","app":"myapp","env":"prod"}

data/lib/tty/logger.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require_relative "logger/config"
+require_relative "logger/data_filter"
 require_relative "logger/event"
 require_relative "logger/formatters/json"
 require_relative "logger/formatters/text"
@@ -17,8 +18,6 @@ module TTY
     # Error raised by this logger
     class Error < StandardError; end
 
-    FILTERED = "[FILTERED]"
-
     LOG_TYPES = {
       debug: { level: :debug },
       info: { level: :info },
@@ -35,7 +34,7 @@ module TTY
     def self.define_level(name, log_level = nil)
       const_level = (LOG_TYPES[name.to_sym] || log_level)[:level]
 
-      loc = caller_locations(0,1)[0]
+      loc = caller_locations(0, 1)[0]
       if loc
         file, line = loc.path, loc.lineno + 7
       else
@@ -95,6 +94,8 @@ module TTY
       @handlers = @config.handlers
       @output = output || @config.output
       @ready_handlers = []
+      @data_filter = DataFilter.new(@config.filters.data,
+                                    mask: @config.filters.mask)
 
       @config.types.each do |name, log_level|
         add_type(name, log_level)
@@ -129,7 +130,7 @@ module TTY
       name = coerce_handler(h)
       global_opts = { output: @output, config: @config }
       opts = global_opts.merge(options)
-      ready_handler = name.new(opts)
+      ready_handler = name.new(**opts)
       @ready_handlers << ready_handler
     end
 
@@ -212,7 +213,8 @@ module TTY
     #   logger.log(:info) { "Deployed successfully" }
     #
     # @api public
-    def log(current_level, *msg, **scoped_fields)
+    def log(current_level, *msg)
+      scoped_fields = msg.last.is_a?(::Hash) ? msg.pop : {}
       fields_copy = scoped_fields.dup
       if msg.empty? && block_given?
         msg = []
@@ -220,8 +222,8 @@ module TTY
           el.is_a?(::Hash) ? fields_copy.merge!(el) : msg << el
         end
       end
-      top_caller = caller_locations(1,1)[0]
-      loc = caller_locations(2,1)[0] || top_caller
+      top_caller = caller_locations(1, 1)[0]
+      loc = caller_locations(2, 1)[0] || top_caller
       label = top_caller.label
       metadata = {
         level: current_level,
@@ -232,7 +234,9 @@ module TTY
         lineno: loc.lineno,
         method: loc.base_label
       }
-      event = Event.new(filter(*msg), @fields.merge(fields_copy), metadata)
+      event = Event.new(filter(*msg),
+                        @data_filter.filter(@fields.merge(fields_copy)),
+                        metadata)
 
       @ready_handlers.each do |handler|
         level = handler.respond_to?(:level) ? handler.level : @config.level
@@ -271,8 +275,8 @@ module TTY
     def filter(*messages)
       messages.reduce([]) do |acc, msg|
         acc << msg.dup.tap do |msg_copy|
-          @config.filters.each do |text, placeholder|
-            msg_copy.gsub!(text, placeholder || FILTERED)
+          @config.filters.message.each do |text|
+            msg_copy.gsub!(text, @config.filters.mask)
           end
         end
         acc

data/lib/tty/logger/config.rb

@@ -5,13 +5,15 @@ require_relative "handlers/console"
 module TTY
   class Logger
     class Config
+      FILTERED = "[FILTERED]"
+
       # The format used for date display
       attr_accessor :date_format
 
       # The format used for time display
       attr_accessor :time_format
 
-      # The storage of placholders to filter sensitive data out from the logs. Defaults to {}
+      # The filters to hide sensitive data from the messages and data.
       attr_accessor :filters
 
       # The format used for displaying structured data
@@ -46,7 +48,6 @@ module TTY
         @max_depth = options.fetch(:max_depth) { 3 }
         @level = options.fetch(:level) { :info }
         @metadata = options.fetch(:metadata) { [] }
-        @filters  = options.fetch(:filters) { {} }
         @handlers = options.fetch(:handlers) { [:console] }
         @formatter = options.fetch(:formatter) { :text }
         @date_format = options.fetch(:date_format) { "%F" }
@@ -55,6 +56,36 @@ module TTY
         @types = options.fetch(:types) { {} }
       end
 
+      class FiltersProvider
+        attr_accessor :message, :data, :mask
+
+        def initialize
+          @message = []
+          @data = []
+          @mask = FILTERED
+        end
+
+        def to_h
+          {message: @message, data: @data, mask: @mask}
+        end
+
+        def to_s
+          to_h.inspect
+        end
+      end
+
+      # The filters to hide sensitive data from the message(s) and data.
+      #
+      # @return [FiltersProvider]
+      #
+      # @api public
+      def filters
+        @filters ||= FiltersProvider.new
+      end
+
+      # Allow to overwirte filters
+      attr_writer :filters
+
       # Clone settings
       #
       # @api public
@@ -83,7 +114,7 @@ module TTY
       def to_h
         {
           date_format: date_format,
-          filters: filters,
+          filters: filters.to_h,
           formatter: formatter,
           handlers: handlers,
           level: level,

data/lib/tty/logger/data_filter.rb

@@ -0,0 +1,121 @@
+# frozen_string_literal: true
+
+module TTY
+  class Logger
+    class DataFilter
+      FILTERED = "[FILTERED]"
+      DOT = "."
+
+      attr_reader :filters, :compiled_filters, :mask
+
+      # Create a data filter instance with filters.
+      #
+      # @example
+      #  TTY::Logger::DataFilter.new(%w[foo], mask: "<SECRET>")
+      #
+      # @param [String] mask
+      #   the mask to replace object with. Defaults to `"[FILTERED]"`
+      #
+      # @api private
+      def initialize(filters = [], mask: nil)
+        @mask = mask || FILTERED
+        @filters = filters
+        @compiled_filters = compile(filters)
+      end
+
+      # Filter object for keys matching provided filters.
+      #
+      # @example
+      #   data_filter = TTY::Logger::DataFilter.new(%w[foo])
+      #   data_filter.filter({"foo" => "bar"})
+      #   # => {"foo" => "[FILTERED]"}
+      #
+      # @param [Object] obj
+      #   the object to filter
+      #
+      # @api public
+      def filter(obj)
+        return obj if filters.empty?
+
+        hash = obj.reduce({}) do |acc, (k, v)|
+          acc[k] = filter_val(k, v)
+          acc
+        end
+        hash
+      end
+
+      private
+
+      def compile(filters)
+        compiled = {
+          regexps: [],
+          nested_regexps: [],
+          blocks: [],
+        }
+        strings = []
+        nested_strings = []
+
+        filters.each do |filter|
+          case filter
+          when Proc
+            compiled[:blocks] << filter
+          when Regexp
+            if filter.to_s.include?(DOT)
+              compiled[:nested_regexps] << filter
+            else
+              compiled[:regexps] << filter
+            end
+          else
+            exp = Regexp.escape(filter)
+            if exp.include?(DOT)
+              nested_strings << exp
+            else
+              strings << exp
+            end
+          end
+        end
+
+        if !strings.empty?
+          compiled[:regexps] << /^(#{strings.join("|")})$/
+        end
+
+        if !nested_strings.empty?
+          compiled[:nested_regexps] << /^(#{nested_strings.join("|")})$/
+        end
+
+        compiled
+      end
+
+      def filter_val(key, val, composite = [])
+        return mask if filtered?(key, composite)
+
+        case val
+        when Hash then filter_obj(key, val, composite << key)
+        when Array then filter_arr(key, val, composite)
+        else val
+        end
+      end
+
+      def filtered?(key, composite)
+        composite_key = composite + [key]
+        joined_key = composite_key.join(DOT)
+        @compiled_filters[:regexps].any? { |reg| !!reg.match(key.to_s) } ||
+          @compiled_filters[:nested_regexps].any? { |reg| !!reg.match(joined_key) } ||
+          @compiled_filters[:blocks].any? { |block| block.(composite_key.dup) }
+      end
+
+      def filter_obj(_key, obj, composite)
+        obj.reduce({}) do |acc, (k, v)|
+          acc[k] = filter_val(k, v, composite)
+          acc
+        end
+      end
+
+      def filter_arr(key, obj, composite)
+        obj.reduce([]) do |acc, v|
+          acc << filter_val(key, v, composite)
+        end
+      end
+    end # DataFilter
+  end # Logger
+end # TTY

data/lib/tty/logger/handlers/console.rb

@@ -57,6 +57,14 @@ module TTY
           }
         }
 
+        TEXT_REGEXP = /([{}()\[\]])?(["']?)(\S+?)(["']?=)/.freeze
+        JSON_REGEXP = /\"([^,]+?)\"(?=:)/.freeze
+
+        COLOR_PATTERNS = {
+          text: [TEXT_REGEXP, ->(c) { "\\1\\2" + c.("\\3") + "\\4" }],
+          json: [JSON_REGEXP, ->(c) { "\"" + c.("\\1") + "\"" }]
+        }.freeze
+
         attr_reader :output
 
         attr_reader :config
@@ -67,6 +75,8 @@ module TTY
                        styles: {})
           @output = Array[output].flatten
           @formatter = coerce_formatter(formatter || config.formatter).new
+          @formatter_name = @formatter.class.name.split("::").last.downcase
+          @color_pattern = COLOR_PATTERNS[@formatter_name.to_sym]
           @config = config
           @styles = styles
           @level = level || @config.level
@@ -108,10 +118,10 @@ module TTY
           fmt << color.(style[:label]) + (" " * style[:levelpad])
           fmt << "%-25s" % event.message.join(" ")
           unless event.fields.empty?
+            pattern, replacement = *@color_pattern
             fmt << @formatter.dump(event.fields, max_bytes: config.max_bytes,
-                                   max_depth: config.max_depth).
-                                   gsub(/(\S+)(?=\=)/, color.("\\1")).
-                                   gsub(/\"([^,]+?)\"(?=:)/, "\"" + color.("\\1") + "\"")
+                                                 max_depth: config.max_depth)
+                             .gsub(pattern, replacement.(color))
           end
           unless event.backtrace.empty?
             fmt << "\n" + format_backtrace(event)