loggerstash 0.0.1

data/README.md

@@ -0,0 +1,150 @@
+Ever wanted to have your loggers send their data direct to logstash, without
+going through a dozen intermediate processing steps?  Do you worry your log
+entries are being dropped because you're forced to use UDP for sending log
+events?  Would you like to have actionable metrics showing exactly how your
+log forwarding system is working?  Do you dream of being able to use modern
+DNS record types, such as SRV (standardised in the year 2000!), to indicate
+where your logstash servers are?
+
+If so, you're in the right place.
+
+
+# Installation
+
+It's a gem:
+
+    gem install loggerstash
+
+There's also the wonders of [the Gemfile](http://bundler.io):
+
+    gem 'loggerstash'
+
+If you're the sturdy type that likes to run from git:
+
+    rake install
+
+Or, if you've eschewed the convenience of Rubygems entirely, then you
+presumably know what to do already.
+
+
+# Logstash Configuration
+
+In order for logstash to receive the events being written, it must have a
+`json_lines` TCP input configured.  Something like this will do the trick:
+
+    input {
+      tcp {
+        id    => "json_lines"
+        port  => 5151
+        codec => "json_lines"
+      }
+    }
+
+
+# Usage
+
+Start by including the necessary file:
+
+    require 'loggerstash'
+
+next, create a new instance of `Loggerstash`, pointing to the logstash
+server you wish to use:
+
+    ls = Loggerstash.new(logstash_server: "192.0.2.42:5151")
+
+Anything that is a valid [logstash_writer
+`server_name`](https://github.com/discourse/logstash_writer#usage) can be
+specified as the `logstash_server`.
+
+Once you have a Loggerstash, you just need to attach your `Loggerstash`
+instance to either the `Logger` class (to have all loggers forward to
+logstash), or to individual instances of Logger that you wish to forward:
+
+    # Forward just this logger's messages to logstash
+    l = Logger.new($stderr)
+    ls.attach(l)
+
+    # Forward all Logger instances' messages to logstash
+    ls.attach(Logger)
+
+    # Forward multiple loggers' messages to logstash
+    some_logger = Logger.new($stderr)
+    another_logger = Logger.new($stderr)
+
+    ls.attach(some_logger)
+    ls.attach(another_logger)
+
+You can, in *theory*, attach Loggerstash to other things, but since it hooks
+into the deep internals of Logger, it's unlikely to work very well on
+anything else.
+
+If you'd like to provide a richer event to logstash than the default (which
+basically just maps progname, timestamp, severity, and message into a hash),
+you can extend the default formatter:
+
+    ls.formatter = ->(s, t, p, m) do
+      default_formatter.call(s, t, p, m).merge(some_tag: "ohai!")
+    end
+
+Of course, you can do entirely your own thing, and not call the default
+formatter at all if you'd prefer.
+
+You can also specify the formatter as a parameter to the constructor:
+
+    ls = Loggerstash.new(logstash_server: "...", formatter: ->(s, t, p, m) { ... }
+
+If you want to expose the metrics maintained by the underlying
+LogstashWriter instance, you can specify a `Prometheus::Client::Registry` to
+register the metrics:
+
+    Loggerstash.new(logstash_server: "...", metrics_registry: @metrics_server.registry)
+
+    # ... OR ...
+    ls.metrics_registry = @metrics_server.registry
+
+You can only pass a given metrics registry to one instance of `Loggerstash`,
+because otherwise the registered metrics will conflict and everything will
+be awful.  If anyone ever comes up with a real-world use-case for needing
+multiple Loggerstashes pointing to the same metrics registry, a PR would be
+accepted to specify a prefix on the registered metrics.
+
+Note that once you have attached a `Loggerstash` instance to a logger, you
+can't change the `Loggerstash` configuration -- calls to any of the
+configuration setters will raise an exception.
+
+
+## Prometheus Metrics
+
+If you specify a metrics registry to use, a set of metrics with a
+`logstash_writer_` prefix will be included and maintained.  For the complete
+set of metrics and their meanings, refer to [the `LogstashWriter`
+documentation](https://github.com/discourse/logstash_writer#prometheus-metrics).
+
+
+# Contributing
+
+Patches can be sent as [a Github pull
+request](https://github.com/discourse/loggerstash).  This project is
+intended to be a safe, welcoming space for collaboration, and contributors
+are expected to adhere to the [Contributor Covenant code of
+conduct](CODE_OF_CONDUCT.md).
+
+
+# Licence
+
+Unless otherwise stated, everything in this repo is covered by the following
+copyright notice:
+
+    Copyright (C) 2018  Civilized Discourse Construction Kit, Inc.
+
+    This program is free software: you can redistribute it and/or modify it
+    under the terms of the GNU General Public License version 3, as
+    published by the Free Software Foundation.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <http://www.gnu.org/licenses/>.

data/lib/loggerstash.rb

@@ -0,0 +1,116 @@
+require 'logstash_writer'
+require 'thread'
+
+# A sidecar class to augment a Logger with super-cow-logstash-forwarding
+# powers.
+#
+class Loggerstash
+  # Base class of all Loggerstash errors
+  #
+  class Error < StandardError; end
+
+  # Raised if any configuration setter methods are called (`Loggerstash#<anything>=`)
+  # after the loggerstash instance has been attached to a logger.
+  #
+  class AlreadyRunningError < Error; end
+
+  attr_writer :formatter
+
+  def initialize(logstash_server:, metrics_registry: nil, formatter: nil)
+    @logstash_server = logstash_server
+    @metrics_registry = metrics_registry
+    @formatter = formatter
+
+    @op_mutex = Mutex.new
+  end
+
+  def attach(obj)
+    @op_mutex.synchronize do
+      obj.instance_variable_set(:@loggerstash, self)
+
+      if obj.is_a?(Module)
+        obj.prepend(Mixin)
+      else
+        obj.singleton_class.prepend(Mixin)
+      end
+
+      run_writer
+    end
+  end
+
+  %i{logstash_server metrics_registry}.each do |sym|
+    define_method(:"#{sym}=") do |v|
+      @op_mutex.synchronize do
+        if @logstash_writer
+          raise AlreadyRunningError,
+                "Cannot change #{sym} once writer is running"
+        end
+        instance_variable_set(:"@#{sym}", v)
+      end
+    end
+  end
+
+  def log_message(s, t, p, m)
+    @op_mutex.synchronize do
+      if @logstash_writer.nil?
+        run_writer
+      end
+
+      @logstash_writer.send_event((@formatter || default_formatter).call(s, t, p, m))
+    end
+  end
+  private
+
+  def run_writer
+    unless @op_mutex.owned?
+      raise RuntimeError,
+            "Must call run_writer while holding @op_mutex"
+    end
+
+    if @logstash_writer.nil?
+      {}.tap do |opts|
+        opts[:server_name] = @logstash_server
+        if @metrics_registry
+          opts[:metrics_registry] = @metrics_registry
+        end
+
+        @logstash_writer = LogstashWriter.new(**opts)
+        @logstash_writer.run
+      end
+    end
+  end
+
+  def default_formatter
+    @default_formatter ||= ->(s, t, p, m) do
+      {
+        "@timestamp" => t.utc.strftime("%FT%T.%NZ"),
+        message: m,
+        severity: s.downcase,
+      }.tap do |ev|
+        ev[:progname] = p if p
+      end
+    end
+  end
+
+  module Mixin
+    private
+
+    # Hooking into this specific method may seem... unorthodox, but
+    # it seemingly has an extremely stable interface and is the most
+    # appropriate place to inject ourselves.
+    def format_message(s, t, p, m)
+      loggerstash.log_message(s, t, p, m)
+
+      super
+    end
+
+    def loggerstash
+      ([self] + self.class.ancestors).find { |m| m.instance_variable_defined?(:@loggerstash) }.instance_variable_get(:@loggerstash).tap do |ls|
+        if ls.nil?
+          raise RuntimeError,
+                "Cannot find loggerstash instance.  CAN'T HAPPEN."
+        end
+      end
+    end
+  end
+end

data/loggerstash.gemspec

@@ -0,0 +1,44 @@
+begin
+  require 'git-version-bump'
+rescue LoadError
+  nil
+end
+
+Gem::Specification.new do |s|
+  s.name = "loggerstash"
+
+  s.version = GVB.version rescue "0.0.0.1.NOGVB"
+  s.date    = GVB.date    rescue Time.now.strftime("%Y-%m-%d")
+
+  s.platform = Gem::Platform::RUBY
+
+  s.summary  = "Monkeypatch Logger to send log entries to logstash"
+  s.description = <<~EOF
+    Provides a module you can prepend into any instance of Logger to cause
+    it to send all logged entries to a Logstash server, in addition to
+    however the message would have been handled otherwise.
+  EOF
+
+  s.authors  = ["Matt Palmer"]
+  s.email    = ["matt.palmer@discourse.org"]
+  s.homepage = "https://github.com/discourse/loggerstash"
+
+  s.files = `git ls-files -z`.split("\0").reject { |f| f =~ /^(G|spec|Rakefile)/ }
+
+  s.required_ruby_version = ">= 2.3.0"
+
+  s.add_runtime_dependency "logstash_writer", "~> 0.0"
+
+  s.add_development_dependency 'bundler'
+  s.add_development_dependency 'github-release'
+  s.add_development_dependency 'git-version-bump'
+  s.add_development_dependency 'guard-rspec'
+  s.add_development_dependency 'guard-rubocop'
+  s.add_development_dependency 'rack-test'
+  s.add_development_dependency 'rake', "~> 12.0"
+  s.add_development_dependency 'redcarpet'
+  s.add_development_dependency 'rspec'
+  s.add_development_dependency 'rubocop'
+  s.add_development_dependency 'simplecov'
+  s.add_development_dependency 'yard'
+end

metadata

@@ -0,0 +1,238 @@
+--- !ruby/object:Gem::Specification
+name: loggerstash
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- Matt Palmer
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2018-10-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: logstash_writer
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: github-release
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: git-version-bump
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: guard-rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: guard-rubocop
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rack-test
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '12.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '12.0'
+- !ruby/object:Gem::Dependency
+  name: redcarpet
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rubocop
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: simplecov
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: yard
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: |
+  Provides a module you can prepend into any instance of Logger to cause
+  it to send all logged entries to a Logstash server, in addition to
+  however the message would have been handled otherwise.
+email:
+- matt.palmer@discourse.org
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".rubocop.yml"
+- ".travis.yml"
+- ".yardopts"
+- CODE_OF_CONDUCT.md
+- CONTRIBUTING.md
+- LICENCE
+- README.md
+- lib/loggerstash.rb
+- loggerstash.gemspec
+homepage: https://github.com/discourse/loggerstash
+licenses: []
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.3.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.7.7
+signing_key: 
+specification_version: 4
+summary: Monkeypatch Logger to send log entries to logstash
+test_files: []