twiglet 3.14.2 → 3.15.0

data/docs/index.md

@@ -1,256 +0,0 @@
-# Twiglet: Ruby version
-Like a log, only smaller.
-
-This library provides a minimal JSON logging interface suitable for use in (micro)services.  See the [RATIONALE](docs/RATIONALE.md) for design rationale and an explantion of the Elastic Common Schema that we are using for log attribute naming.
-
-## Installation
-
-```bash
-gem install twiglet
-```
-
-## How to use
-
-### Instantiate the logger
-
-```ruby
-require 'twiglet/logger'
-logger = Twiglet::Logger.new('service name')
-```
-#### Optional initialization parameters
-A hash can optionally be passed in as a keyword argument for `default_properties`. This hash must be in the Elastic Common Schema format and will be present in every log message created by this Twiglet logger object.
-
-You may also provide an optional `output` keyword argument which should be an object with a `puts` method - like `$stdout`.
-
-In addition, you can provide another optional keyword argument called `now`, which should be a function returning a `Time` string in ISO8601 format.
-
-Lastly, you may provide the optional keyword argument `level` to initialize the logger with a severity threshold. Alternatively, the threshold can be updated at runtime by calling the `level` instance method.
-
-The defaults for both `output` and `now` should serve for most uses, though you may want to override them for testing as we have done [here](test/logger_test.rb).
-
-### Invoke the Logger
-
-```ruby
-logger.error({ event: { action: 'startup' }, message: "Emergency! There's an Emergency going on" })
-```
-
-This will write to STDOUT a JSON string:
-
-```json
-{"service":{"name":"service name"},"@timestamp":"2020-05-14T10:54:59.164+01:00","log":{"level":"error"},"event":{"action":"startup"},"message":"Emergency! There's an Emergency going on"}
-```
-
-Obviously the timestamp will be different.
-
-Alternatively, if you just want to log some error string:
-
-```ruby
-logger.error("Emergency! There's an Emergency going on")
-```
-
-This will write to STDOUT a JSON string:
-
-```json
-{"service":{"name":"service name"},"@timestamp":"2020-05-14T10:54:59.164+01:00","log":{"level":"error"}, "message":"Emergency! There's an Emergency going on"}
-```
-
-A message is always required unless a block is provided. The message can be an object or a string.
-
-#### Error logging
-An optional error can also be provided, in which case the error message and backtrace will be logged in the relevant ECS compliant fields:
-
-```ruby
-db_err = StandardError.new('Connection timed-out')
-logger.error({ message: 'DB connection failed.' }, db_err)
-
-# this is also valid
-logger.error('DB connection failed.', db_err)
-```
-
-These will both result in the same JSON string written to STDOUT:
-
-```json
-{"ecs":{"version":"1.5.0"},"@timestamp":"2020-08-21T15:44:37.890Z","service":{"name":"service name"},"log":{"level":"error"},"message":"DB connection failed.","error":{"message":"Connection timed-out"}}
-```
-
-#### Custom fields
-Log custom event-specific information simply as attributes in a hash:
-
-```ruby
-logger.info({
-  event: { action: 'HTTP request' },
-  message: 'GET /pets success',
-  trace: { id: '1c8a5fb2-fecd-44d8-92a4-449eb2ce4dcb' },
-  http: {
-    request: { method: 'get' },
-    response: { status_code: 200 }
-  },
-  url: { path: '/pets' }
-})
-```
-
-This writes:
-
-```json
-{"service":{"name":"service name"},"@timestamp":"2020-05-14T10:56:49.527+01:00","log":{"level":"info"},"event":{"action":"HTTP request"},"message":"GET /pets success","trace":{"id":"1c8a5fb2-fecd-44d8-92a4-449eb2ce4dcb"},"http":{"request":{"method":"get"},"response":{"status_code":200}},"url":{"path":"/pets"}}
-```
-
-Similar to error you can use string logging here as:
-
-```
-logger.info('GET /pets success')
-```
-
-This writes:
-
-```json
-{"service":{"name":"service name"},"@timestamp":"2020-05-14T10:56:49.527+01:00","log":{"level":"info"}}
-```
-
-It may be that when making a series of logs that write information about a single event, you may want to avoid duplication by creating an event specific logger that includes the context:
-
-```ruby
-request_logger = logger.with({ event: { action: 'HTTP request'}, trace: { id: '1c8a5fb2-fecd-44d8-92a4-449eb2ce4dcb' }})
-```
-
-This can be used like any other Logger instance:
-
-```ruby
-request_logger.error({
-    message: 'Error 500 in /pets/buy',
-    http: {
-        request: { method: 'post', 'url.path': '/pet/buy' },
-        response: { status_code: 500 }
-    }
-})
-```
-
-which will print:
-
-```json
-{"service":{"name":"service name"},"@timestamp":"2020-05-14T10:58:30.780+01:00","log":{"level":"error"},"event":{"action":"HTTP request"},"trace":{"id":"126bb6fa-28a2-470f-b013-eefbf9182b2d"},"message":"Error 500 in /pets/buy","http":{"request":{"method":"post","url.path":"/pet/buy"},"response":{"status_code":500}}}
-```
-
-### Log formatting
-Some third party applications will allow you to optionally specify a [log formatter](https://ruby-doc.org/stdlib-2.4.0/libdoc/logger/rdoc/Logger/Formatter.html).
-Supplying a Twiglet log formatter will format those third party logs so that they are ECS compliant and have the same default parameters as your application's internal logs.
-
-To access the formatter:
-```ruby
-logger.formatter
-```
-
-### HTTP Request Logging
-Take a look at this sample [Rack application](examples/rack/example_rack_app.rb#L15) with an ECS compliant
-[request logger](/examples/rack/request_logger.rb) as a template when configuring your own request logging middleware with Twiglet.
-
-### Log format validation
-Twiglet allows for the configuration of a custom validation schema. The validation schema must be [JSON Schema](https://json-schema.org/) compliant. Any fields not explicitly included in the provided schema are permitted by default.
-
-For example, given the following JSON Schema:
-```ruby
-validation_schema = <<-JSON
-    {
-      "type": "object",
-      "required": ["pet"],
-      "properties": {
-        "pet": {
-          "type": "object",
-          "required": ["name", "best_boy_or_girl?"],
-          "properties": {
-            "name": {
-              "type": "string",
-              "minLength": 1
-            },
-            "good_boy?": {
-              "type": "boolean"
-            }
-          }
-        }
-      }
-    }
-JSON
-```
-
-The logger can be instantiated with the custom schema
-```ruby
-custom_logger = Twiglet::Logger.new('service name', validation_schema: validation_schema)
-```
-
-Compliant log messages will log as normal.
-```ruby
-# this is compliant
-custom_logger.debug(pet: { name: 'Davis', good_boy?: true })
-
-# the result
-{:ecs=>{:version=>"1.5.0"}, :@timestamp=>"2020-05-11T15:01:01.000Z", :service=>{:name=>"petshop"}, :log=>{:level=>"debug"}, :pet=>{:name=>"Davis", :good_boy?=>true}}
-```
-
-Non compliant messages will raise an error.
-```ruby
-begin
-  custom_logger.debug(pet: { name: 'Davis' })
-rescue JSON::Schema::ValidationError
-  # we forgot to specify that he's a good boy!
-  puts 'uh-oh'
-end
-```
-
-#### Customizing error responses
-Depending on the application, it may not be desirable for the logger to raise Runtime errors. Twiglet allows you to configure a custom response for handling validation errors.
-
-Configure error handling by writing a block
-```ruby
-logger.configure_validation_error_response do |error|
-  # validation error handling goes here
-  # for example:
-  {YOUR APPLICATION BUG TRACKING SERVICE}.notify_error(error)
-end
-
-```
-
-### Use of dotted keys (DEPRECATED)
-
-Writing nested json objects could be confusing. This library has a built-in feature to convert dotted keys into nested objects, so if you log like this:
-
-```ruby
-logger.info({
-    'event.action': 'HTTP request',
-    message: 'GET /pets success',
-    'trace.id': '1c8a5fb2-fecd-44d8-92a4-449eb2ce4dcb',
-    'http.request.method': 'get',
-    'http.response.status_code': 200,
-    'url.path': '/pets'
-})
-```
-
-or mix between dotted keys and nested objects:
-
-```ruby
-logger.info({
-    'event.action': 'HTTP request',
-    message: 'GET /pets success',
-    trace: { id: '1c8a5fb2-fecd-44d8-92a4-449eb2ce4dcb' },
-    'http.request.method': 'get',
-    'http.response.status_code': 200,
-    url: { path: '/pets' }
-})
-```
-
-Both cases would print out exact the same log item:
-
-```json
-{"service":{"name":"service name"},"@timestamp":"2020-05-14T10:59:31.183+01:00","log":{"level":"info"},"event":{"action":"HTTP request"},"message":"GET /pets success","trace":{"id":"1c8a5fb2-fecd-44d8-92a4-449eb2ce4dcb"},"http":{"request":{"method":"get"},"response":{"status_code":200}},"url":{"path":"/pets"}}
-```
-
-## How to contribute
-
-First: Please read our project [Code of Conduct](../CODE_OF_CONDUCT.md).
-
-Second: run the tests and make sure your changes don't break anything:
-
-```bash
-bundle exec rake test
-```
-
-Then please feel free to submit a PR.

data/example_app.rb

@@ -1,60 +0,0 @@
-# frozen_string_literal: true
-
-require_relative 'lib/twiglet/logger'
-
-PORT = 8080
-
-logger = Twiglet::Logger.new('petshop')
-
-# Start our petshop
-logger.info(
-  {
-    event: {
-      action: 'startup'
-    },
-    message: "Ready to go, listening on port #{PORT}",
-    server: {
-      port: PORT
-    }
-  }
-)
-
-# Use text logging
-logger.info("Ready to go, listening on port #{PORT}")
-#
-# We get a request
-request_logger = logger.with(
-  {
-    event: {
-      action: 'HTTP request'
-    },
-    trace: {
-      id: '126bb6fa-28a2-470f-b013-eefbf9182b2d'
-    }
-  }
-)
-
-# Oh noes!
-db_err = StandardError.new('Connection timed-out')
-
-request_logger.error({ message: 'DB connection failed.' }, db_err) if db_err
-
-# We return an error to the requester
-request_logger.info(
-  {
-    message: 'Internal Server Error',
-    http: {
-      request: {
-        method: 'get'
-      },
-      response: {
-        status_code: 500
-      }
-    }
-  }
-)
-
-# Logging with an empty message is an anti-pattern and is therefore forbidden
-# Both of the following lines would throw an error
-# request_logger.error({ message: "" })
-# logger.debug({ message: " " })

data/examples/rack/example_rack_app.rb

@@ -1,17 +0,0 @@
-require 'twiglet/logger'
-require 'request_logger'
-
-# basic rack application
-class Application
-  def call(_env)
-    status  = 200
-    headers = { "Content-Type" => "text/json" }
-    body    = ["Example rack app"]
-
-    [status, headers, body]
-  end
-end
-
-use RequestLogger, Twiglet::Logger.new('example_app')
-
-run Application.new

data/examples/rack/request_logger.rb

@@ -1,66 +0,0 @@
-# Middleware for logging request logs
-class RequestLogger
-  def initialize(app, logger)
-    @app = app
-    @logger = logger
-  end
-
-  def call(env)
-    status, headers, body = @app.call(env)
-    log(env, status)
-    [status, headers, body]
-  rescue StandardError => e
-    log_error(env, 500, e)
-    raise e
-  end
-
-  private
-
-  def log(env, status)
-    fields = get_fields(env, status)
-    @logger.info(fields)
-  end
-
-  def log_error(env, status, error)
-    fields = get_fields(env, status)
-    @logger.error(fields, error)
-  end
-
-  # https://www.elastic.co/guide/en/ecs/1.5/ecs-field-reference.html
-  def get_fields(env, status)
-    message = "#{env['REQUEST_METHOD']}: #{env['PATH_INFO']}"
-
-    {
-      http: http_fields(env, status),
-      url: url_fields(env),
-      client: {
-        ip: env['HTTP_TRUE_CLIENT_IP'] || env['REMOTE_ADDR']
-      },
-      user_agent: {
-        original: env['HTTP_USER_AGENT']
-      },
-      message: message
-    }
-  end
-
-  def http_fields(env, status)
-    {
-      request: {
-        method: env['REQUEST_METHOD'],
-        mime_type: env['HTTP_ACCEPT']
-      },
-      response: {
-        status: status
-      },
-      version: env['HTTP_VERSION']
-    }
-  end
-
-  def url_fields(env)
-    {
-      path: env['PATH_INFO'],
-      query: env['QUERY_STRING'],
-      domain: env['SERVER_NAME']
-    }
-  end
-end

data/examples/rack/request_logger_test.rb

@@ -1,91 +0,0 @@
-require 'minitest/autorun'
-require_relative '../../lib/twiglet/logger'
-require_relative './request_logger'
-require 'rack'
-
-describe RequestLogger do
-  let(:output) { StringIO.new }
-
-  before { output.rewind }
-
-  it 'log should not be empty' do
-    request.get("/some/path")
-    log = output.string
-    refute_empty log
-  end
-
-  it 'logs the request data' do
-    request.get(
-      "/some/path?some_var=1", 'HTTP_ACCEPT' => 'application/json',
-                               'REMOTE_ADDR' => '0.0.0.0',
-                               'HTTP_VERSION' => 'HTTP/1.1',
-                               'HTTP_USER_AGENT' => 'Mozilla/5.0 (Macintosh)'
-    )
-    log = JSON.parse(output.string)
-
-    expected_log = {
-      "log" => { "level" => "info" },
-      "http" => {
-        "request" => {
-          "method" => "GET",
-          "mime_type" => 'application/json'
-        },
-        "response" => {
-          "status" => 200
-        },
-        "version" => 'HTTP/1.1'
-      },
-      "url" => {
-        "path" => "/some/path",
-        "query" => "some_var=1",
-        "domain" => "example.org"
-      },
-      "client" => {
-        'ip' => '0.0.0.0'
-      },
-      "user_agent" => {
-        "original" => 'Mozilla/5.0 (Macintosh)'
-      },
-      "message" => "GET: /some/path"
-    }
-
-    assert_equal(log['log'], expected_log['log'])
-    assert_equal(log['http'], expected_log['http'])
-    assert_equal(log['url'], expected_log['url'])
-    assert_equal(log['user_agent'], expected_log['user_agent'])
-    assert_equal(log['message'], expected_log['message'])
-  end
-
-  it 'does not log PII' do
-    request.post("/user/info", input_data: { credit_card_no: '1234' })
-    log = output.string
-    assert_includes log, "POST: /user/info"
-    refute_includes log, 'credit_card_no'
-    refute_includes log, '1234'
-  end
-
-  it 'logs an error message when a request is bad' do
-    expect { bad_request.get("/some/path") }.must_raise StandardError
-    log = JSON.parse(output.string)
-    assert_equal log['log']['level'], 'error'
-    assert_equal log['error']['message'], 'some exception'
-    assert_equal log['error']['type'], 'StandardError'
-    assert_includes log['error']['stack_trace'].first, 'examples/rack/request_logger_test.rb'
-  end
-end
-
-def request
-  app = ->(env) { [200, env, "app"] }
-  base_request(app)
-end
-
-def bad_request
-  app = Rack::Lint.new ->(_env) { raise StandardError, 'some exception' }
-  base_request(app)
-end
-
-def base_request(app)
-  logger = Twiglet::Logger.new('example', output: output)
-  req_logger = RequestLogger.new(app, logger)
-  Rack::MockRequest.new(req_logger)
-end

data/test/error_serialiser_test.rb

@@ -1,18 +0,0 @@
-# frozen_string_literal: true
-
-require 'minitest/autorun'
-require 'minitest/mock'
-require_relative '../lib/twiglet/error_serialiser'
-
-describe Twiglet::ErrorSerialiser do
-  describe 'logging an exception' do
-    it 'should log an error with backtrace' do
-      1 / 0
-    rescue StandardError => e
-      error_hash = Twiglet::ErrorSerialiser.new.serialise_error(e)
-      assert_equal 'divided by 0', error_hash[:error][:message]
-      assert_equal 'ZeroDivisionError', error_hash[:error][:type]
-      assert_match 'test/error_serialiser_test.rb', error_hash[:error][:stack_trace].first
-    end
-  end
-end

data/test/formatter_test.rb

@@ -1,89 +0,0 @@
-# frozen_string_literal: true
-
-require 'minitest/autorun'
-require 'json'
-require_relative '../lib/twiglet/formatter'
-require_relative '../lib/twiglet/validator'
-
-describe Twiglet::Formatter do
-  before do
-    @now = -> { Time.utc(2020, 5, 11, 15, 1, 1) }
-    @formatter = Twiglet::Formatter.new('petshop', now: @now, validator: Twiglet::Validator.new({}.to_json))
-  end
-
-  it 'initializes an instance of a Ruby Logger Formatter' do
-    assert @formatter.is_a?(::Logger::Formatter)
-  end
-
-  it 'returns a formatted log from a string message' do
-    msg = @formatter.call('warn', nil, nil, 'shop is running low on dog food')
-    expected_log = {
-      "ecs" => {
-        "version" => '1.5.0'
-      },
-      "@timestamp" => '2020-05-11T15:01:01.000Z',
-      "service" => {
-        "name" => 'petshop'
-      },
-      "log" => {
-        "level" => 'warn'
-      },
-      "message" => 'shop is running low on dog food'
-    }
-    assert_equal JSON.parse(msg), expected_log
-  end
-
-  it 'merges the outputs of the context provider into messages logs' do
-    provider = -> { { 'request' => { 'id' => '1234567890' } } }
-    formatter = Twiglet::Formatter.new(
-      'petshop', now: @now, validator: Twiglet::Validator.new({}.to_json),
-                 context_provider: provider
-    )
-    msg = formatter.call('warn', nil, nil, 'shop is running low on dog food')
-    expected_log = {
-      "ecs" => {
-        "version" => '1.5.0'
-      },
-      "@timestamp" => '2020-05-11T15:01:01.000Z',
-      "service" => {
-        "name" => 'petshop'
-      },
-      "log" => {
-        "level" => 'warn'
-      },
-      "message" => 'shop is running low on dog food',
-      "request" => {
-        'id' => '1234567890'
-      }
-    }
-    assert_equal JSON.parse(msg), expected_log
-  end
-
-  it 'merges the outputs of all context providers into the messages log' do
-    provider_1 = -> { { 'request' => { 'id' => '1234567890' } } }
-    provider_2 = -> { { 'request' => { 'type' => 'test' } } }
-    formatter = Twiglet::Formatter.new(
-      'petshop', now: @now, validator: Twiglet::Validator.new({}.to_json),
-                 context_providers: [provider_1, provider_2]
-    )
-    msg = formatter.call('warn', nil, nil, 'shop is running low on dog food')
-    expected_log = {
-      "ecs" => {
-        "version" => '1.5.0'
-      },
-      "@timestamp" => '2020-05-11T15:01:01.000Z',
-      "service" => {
-        "name" => 'petshop'
-      },
-      "log" => {
-        "level" => 'warn'
-      },
-      "message" => 'shop is running low on dog food',
-      "request" => {
-        'id' => '1234567890',
-        'type' => 'test'
-      }
-    }
-    assert_equal expected_log, JSON.parse(msg)
-  end
-end