fluent-plugin-systemd 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 07f12e01a1d81c731f44477a051807042230adfa
-  data.tar.gz: 8717ea703cd39fb947af86b5773a68de5b55c040
+  metadata.gz: 14abb1b9c6b985bbaee61aae73791108578c738e
+  data.tar.gz: ab54b3167ad1832efd73d15eff17d0b4ed2d6e0a
 SHA512:
-  metadata.gz: 92b3275665979af1a9d691f05acdc65a88fd88b062c6224673cde0c691c9c7a4376a1c97c96c7e1dc9cc54f224fe27e7c39600bc0b6307c88ad182e442971520
-  data.tar.gz: 110c2d6ff7ebeac0eb96ceaf0d470d5bdfdf5da2e4556f8b2fc98b213e08b87553d231073f2828a5050dc88437f307e2846cea0ebfd39c131465a34b83123e5e
+  metadata.gz: 26dca0c3b93b49899fbb903ccfc66f79d9f2ded5416ce1bdd28b920b83583a4de29c812a426aabe9aad4b5718055b532f2fcaa6c2b15d77e4ea8bcb46fcebe23
+  data.tar.gz: a715f762f61481b93c1179af10c30f978cbafad7c01ffdd136a79e6d21757d5ce449f59640887d875fc74327c1c1c51d2d09a08eee1a0f1760054ecfc111c9e7

data/README.md

@@ -1,45 +1,57 @@
-# systemd input plugin for [Fluentd](http://github.com/fluent/fluentd)
+# systemd plugin for [Fluentd](http://github.com/fluent/fluentd)
 
 [![Build Status](https://travis-ci.org/reevoo/fluent-plugin-systemd.svg?branch=master)](https://travis-ci.org/reevoo/fluent-plugin-systemd) [![Code Climate GPA](https://codeclimate.com/github/reevoo/fluent-plugin-systemd/badges/gpa.svg)](https://codeclimate.com/github/reevoo/fluent-plugin-systemd) [![Gem Version](https://badge.fury.io/rb/fluent-plugin-systemd.svg)](https://rubygems.org/gems/fluent-plugin-systemd)
 
-# Requirements <a name="requirements"></a>
+## Overview
+
+**systemd** input plugin reads logs from the systemd journal  
+**systemd** filter plugin allows for basic manipulation of systemd journal entries
+
+## Support
+
+[![Fluentd Slack](http://slack.fluentd.org/badge.svg)](http://slack.fluentd.org/)
+
+Join the #plugin-systemd channel on the [Fluentd Slack](http://slack.fluentd.org/)
 
 
+## Requirements
+
 |fluent-plugin-systemd|fluentd|td-agent|ruby|
 |----|----|----|----|
-| 0.2.x | >= 0.14.11, < 2 | 3 | >= 2.1 |
+| > 0.1.0 | >= 0.14.11, < 2 | 3 | >= 2.1 |
 | 0.0.x | ~> 0.12.0       | 2 | >= 1.9  |
 
-* The 0.1.x series is developed from this branch (master)
+* The 0.x.x series is developed from this branch (master)
 * The 0.0.x series (compatible with fluentd v0.12, and td-agent 2) is developed on the [0.0.x branch](https://github.com/reevoo/fluent-plugin-systemd/tree/0.0.x)
 
-## Overview
-
-**systemd** input plugin reads logs from the systemd journal
-
 ## Installation
 
 Simply use RubyGems:
 
-    gem install fluent-plugin-systemd -v 0.2.0
+    gem install fluent-plugin-systemd -v 0.3.0
 
 or
 
-    td-agent-gem install fluent-plugin-systemd -v 0.2.0
+    td-agent-gem install fluent-plugin-systemd -v 0.3.0
 
-## Configuration
+## Input Plugin Configuration
 
     <source>
       @type systemd
+      tag kube-proxy
       path /var/log/journal
       filters [{ "_SYSTEMD_UNIT": "kube-proxy.service" }]
+      read_from_head true
       <storage>
         @type local
         persistent true
         path kube-proxy.pos
       </storage>
-      tag kube-proxy
-      read_from_head true
+      <entry>
+        field_map {"MESSAGE": "log", "_PID": ["process", "pid"], "_CMDLINE": "process", "_COMM": "cmd"}
+        fields_strip_underscores true
+        fields_lowercase true
+      </entry>
     </source>
 
     <match kube-proxy>
@@ -78,19 +90,83 @@ If true reads all available journal from head, otherwise starts reading from tai
 
 **`strip_underscores`**
 
+_This parameter is deprecated and will be removed in favour of entry in v1.0._
+
 If true strips underscores from the beginning of systemd field names.
 May be useful if outputting to kibana, as underscore prefixed fields are unindexed there.
 
+**`entry`**
+
+Optional configuration for an embeded systemd entry filter. See the  [Filter Plugin Configuration](#filter-plugin-configuration) for config reference.
+
 **`tag`**
 
-_Required_ 
+_Required_
 
 A tag that will be added to events generated by this input.
 
-## Example
+### Example
 
 For an example of a full working setup including the plugin, [take a look at](https://github.com/assemblyline/fluentd)
 
+## Filter Plugin Configuration
+
+    <filter kube-proxy>
+      @type systemd_entry
+      field_map {"MESSAGE": "log", "_PID": ["process", "pid"], "_CMDLINE": "process", "_COMM": "cmd"}
+      field_map_strict false
+      fields_lowercase true
+      fields_strip_underscores true
+    </filter>
+
+**`field_map`**
+
+Object / hash defining a mapping of source fields to destination fields. Destination fields may be existing or new user-defined fields. If multiple source fields are mapped to the same destination field, the contents of the fields will be appended to the destination field in the order defined in the mapping. A field map declaration takes the form of:
+
+    {
+      "<src_field1>": "<dst_field1>",
+      "<src_field2>": ["<dst_field1>", "<dst_field2>"],
+      ...
+    }
+Defaults to an empty map.
+
+**`field_map_strict`**
+
+If true, only destination fields from `field_map` are included in the result. Defaults to false.
+
+**`fields_lowercase`**
+
+If true, lowercase all non-mapped fields. Defaults to false.
+
+**`fields_strip_underscores`**
+
+If true, strip leading underscores from all non-mapped fields. Defaults to false.
+
+### Example
+
+Given a systemd journal source entry:
+```
+{
+  "_MACHINE_ID": "bb9d0a52a41243829ecd729b40ac0bce"
+  "_HOSTNAME": "arch"
+  "MESSAGE": "this is a log message",
+  "_PID": "123"
+  "_CMDLINE": "login -- root"
+  "_COMM": "login"
+}
+```
+The resulting entry using the above sample configuration:
+```
+{
+  "machine_id": "bb9d0a52a41243829ecd729b40ac0bce"
+  "hostname": "arch",
+  "msg": "this is a log message",
+  "pid": "123"
+  "cmd": "login"
+  "process": "123 login -- root"
+}
+```
+
 ## Dependencies
 
 This plugin depends on libsystemd

data/lib/fluent/plugin/filter_systemd_entry.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+require "fluent/plugin/filter"
+require "fluent/plugin/systemd/entry_mutator"
+
+module Fluent
+  module Plugin
+    # Fluentd systemd/journal filter plugin
+    class SystemdEntryFilter < Filter
+      Fluent::Plugin.register_filter("systemd_entry", self)
+
+      config_param :field_map, :hash, default: {}
+      config_param :field_map_strict, :bool, default: false
+      config_param :fields_strip_underscores, :bool, default: false
+      config_param :fields_lowercase, :bool, default: false
+
+      def configure(conf)
+        super
+        @mutator = SystemdEntryMutator.new(**@config_root_section.to_h)
+        if @mutator.field_map_strict && @mutator.field_map.empty?
+          log.warn("`field_map_strict` set to true with empty `field_map`, expect no fields")
+        end
+      end
+
+      def filter(_tag, _time, entry)
+        @mutator.run(entry)
+      end
+    end
+  end
+end

data/lib/fluent/plugin/in_systemd.rb

@@ -1,6 +1,8 @@
+# frozen_string_literal: true
 require "systemd/journal"
 require "fluent/plugin/input"
 require "fluent/plugin/systemd/pos_writer"
+require "fluent/plugin/systemd/entry_mutator"
 
 module Fluent
   module Plugin
@@ -9,13 +11,14 @@ module Fluent
 
       helpers :timer, :storage
 
-      DEFAULT_STORAGE_TYPE = "local".freeze
+      DEFAULT_STORAGE_TYPE = "local"
 
       config_param :path, :string, default: "/var/log/journal"
       config_param :filters, :array, default: []
       config_param :pos_file, :string, default: nil, deprecated: "Use <storage> section with `persistent: true' instead"
       config_param :read_from_head, :bool, default: false
-      config_param :strip_underscores, :bool, default: false
+      config_param :strip_underscores, :bool, default: false, deprecated: "Use <entry> section or `systemd_entry` " \
+                                                                          "filter plugin instead"
       config_param :tag, :string
 
       config_section :storage do
@@ -24,10 +27,24 @@ module Fluent
         config_set_default :persistent, false
       end
 
+      config_section :entry, param_name: "entry_opts", required: false, multi: false do
+        config_param :field_map, :hash, default: {}
+        config_param :field_map_strict, :bool, default: false
+        config_param :fields_strip_underscores, :bool, default: false
+        config_param :fields_lowercase, :bool, default: false
+      end
+
       def configure(conf)
         super
+        @journal = nil
         @pos_storage = PosWriter.new(@pos_file, storage_create(usage: "positions"))
-        @journal    = nil
+        # legacy strip_underscores backwards compatibility (legacy takes
+        # precedence and is mutually exclusive with the entry block)
+        mut_opts = @strip_underscores ? { fields_strip_underscores: true } : @entry_opts.to_h
+        @mutator = SystemdEntryMutator.new(**mut_opts)
+        if @mutator.field_map_strict && @mutator.field_map.empty?
+          log.warn("`field_map_strict` set to true with empty `field_map`, expect no fields")
+        end
       end
 
       def start
@@ -89,17 +106,24 @@ module Fluent
         return unless @journal || init_journal
         init_journal if @journal.wait(0) == :invalidate
         watch do |entry|
-          begin
-            router.emit(@tag, Fluent::EventTime.from_time(entry.realtime_timestamp), formatted(entry))
-          rescue => e
-            log.error("Exception emitting record: #{e}")
-          end
+          emit(entry)
         end
       end
 
+      def emit(entry)
+        router.emit(@tag, Fluent::EventTime.from_time(entry.realtime_timestamp), formatted(entry))
+      rescue Fluent::Plugin::Buffer::BufferOverflowError => e
+        retries ||= 0
+        raise e if retries > 10
+        retries += 1
+        sleep 1.5**retries + rand(0..3)
+        retry
+      rescue => e
+        log.error("Exception emitting record: #{e}")
+      end
+
       def formatted(entry)
-        return entry.to_h unless @strip_underscores
-        Hash[entry.to_h.map { |k, v| [k.gsub(/\A_+/, ""), v] }]
+        @mutator.run(entry)
       end
 
       def watch

data/lib/fluent/plugin/systemd/entry_mutator.rb

@@ -0,0 +1,151 @@
+# frozen_string_literal: true
+require "fluent/config/error"
+
+module Fluent
+  module Plugin
+    # A simple stand-alone configurable mutator for systemd journal entries.
+    #
+    # Note regarding field mapping:
+    # The input `field_map` option is meant to have a structure that is
+    # intuative or logical for humans when declaring a field map.
+    # {
+    #   "<source_field1>" => "<new_field1>",
+    #   "<source_field2>" => ["<new_field1>", "<new_field2>"]
+    # }
+    # Internally the inverse of the human-friendly field_map is
+    # computed (and cached) upon object creation and used as a "mapped model"
+    # {
+    #   "<new_field1>" => ["<source_field1>", "<source_field2>"],
+    #   "<new_field2>" => ["<source_field2>"]
+    # }
+    class SystemdEntryMutator
+
+      Options = Struct.new(
+        :field_map,
+        :field_map_strict,
+        :fields_lowercase,
+        :fields_strip_underscores,
+      )
+
+      def self.default_opts
+        Options.new({}, false, false, false)
+      end
+
+      # Constructor keyword options (all other kwargs are ignored):
+      # field_map - hash describing the desired field mapping in the form:
+      #             {"<source_field>" => "<new_field>", ...}
+      #             where `new_field` is a string or array of strings
+      # field_map_strict - boolean if true will only include new fields
+      #                    defined in `field_map`
+      # fields_strip_underscores - boolean if true will strip all leading
+      #                            underscores from non-mapped fields
+      # fields_lowercase - boolean if true lowercase all non-mapped fields
+      #
+      # raises `Fluent::ConfigError` for invalid options
+      def initialize(**options)
+        @opts = options_from_hash(options)
+        validate_options(@opts)
+        @map = invert_field_map(@opts.field_map)
+        @map_src_fields = @opts.field_map.keys
+        @no_transform = @opts == self.class.default_opts
+      end
+
+      # Expose config state as read-only instance properties of the mutator.
+      def method_missing(sym, *args)
+        return @opts[sym] if @opts.members.include?(sym)
+        super
+      end
+
+      # The main run method that performs all configured mutations, if any,
+      # against a single journal entry. Returns the mutated entry hash.
+      # entry - hash or `Systemd::Journal:Entry`
+      def run(entry)
+        return entry.to_h if @no_transform
+        return map_fields(entry) if @opts.field_map_strict
+        format_fields(entry, map_fields(entry))
+      end
+
+      # Run field mapping against a single journal entry. Returns the mutated
+      # entry hash.
+      # entry - hash or `Systemd::Journal:Entry`
+      def map_fields(entry)
+        mapped = {}
+        @map.each do |cstm, sysds|
+          vals = sysds.collect { |fld| entry[fld] }.compact
+          next if vals.empty? # systemd field does not exist in source entry
+          mapped[cstm] = vals.length == 1 ? vals[0] : vals.join(" ")
+        end
+        mapped
+      end
+
+      # Run field formatting (mutations applied to all non-mapped fields)
+      # against a single journal entry. Returns the mutated entry hash.
+      # entry - hash or `Systemd::Journal:Entry`
+      # mapped - Optional hash that represents a previously mapped entry to
+      #          which the formatted fields will be added
+      def format_fields(entry, mapped = nil)
+        mapped ||= {}
+        entry.each do |fld, val|
+          # don't mess with explicitly mapped fields
+          next if @map_src_fields.include?(fld)
+          fld = fld.gsub(/\A_+/, "") if @opts.fields_strip_underscores
+          fld = fld.downcase if @opts.fields_lowercase
+          # account for mapping (appending) to an existing systemd field
+          mapped[fld] = mapped.key?(fld) ? [val, mapped[fld]].join(" ") : val
+        end
+        mapped
+      end
+
+      private
+
+      # Returns a `SystemdEntryMutator::Options` struct derived from the
+      # elements in the supplied hash merged with the option defaults
+      def options_from_hash(opts)
+        merged = self.class.default_opts
+        merged.each_pair do |k, _|
+          merged[k] = opts[k] if opts.key?(k)
+        end
+        merged
+      end
+
+      def validate_options(opts)
+        unless validate_strings_or_empty(opts[:field_map].keys)
+          err = "`field_map` keys must be strings"
+        end
+        unless validate_strings_or_empty(opts[:field_map].values, true)
+          err = "`field_map` values must be strings or array of strings"
+        end
+        %i[field_map_strict fields_strip_underscores fields_lowercase].each do |opt|
+          err = "`#{opt}` must be boolean" unless [true, false].include?(opts[opt])
+        end
+        fail Fluent::ConfigError, err unless err.nil?
+      end
+
+      # Validates that values in array `arr` are strings. If `nested` is true
+      # also allow and validate that `arr` values can be an array of strings
+      def validate_strings_or_empty(arr, nested = false)
+        return true if arr.empty?
+        arr.each do |v|
+          return true if v.is_a?(String)
+          if v.is_a?(Array) && nested
+            v.each { |nstd| return false unless nstd.is_a?(String) }
+          end
+        end
+        false
+      end
+
+      # Compute the inverse of a human friendly field map `fm` which is what
+      # the mutator uses for the actual mapping. The resulting structure for
+      # the inverse field map hash is:
+      # {"<new_field_name>" => ["<source_field_name>", ...], ...}
+      def invert_field_map(fm)
+        invs = {}
+        fm.values.flatten.uniq.each do |cstm|
+          sysds = fm.select { |_, v| (v == cstm || v.include?(cstm)) }
+          invs[cstm] = sysds.keys
+        end
+        invs
+      end
+    end
+  end
+end

data/lib/fluent/plugin/systemd/pos_writer.rb

@@ -1,3 +1,4 @@
+# frozen_string_literal: true
 require "fluent/plugin/input"
 
 module Fluent
@@ -66,7 +67,7 @@ module Fluent
         def write_pos
           @lock.synchronize do
             if @written_cursor != @cursor
-              file = File.open(@path, "w+", 0644)
+              file = File.open(@path, "w+", 0o644)
               file.print @cursor
               file.close
               @written_cursor = @cursor

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-systemd
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
 platform: ruby
 authors:
 - Ed Robinson
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-02-20 00:00:00.000000000 Z
+date: 2017-07-25 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -109,7 +109,9 @@ extra_rdoc_files: []
 files:
 - LICENCE
 - README.md
+- lib/fluent/plugin/filter_systemd_entry.rb
 - lib/fluent/plugin/in_systemd.rb
+- lib/fluent/plugin/systemd/entry_mutator.rb
 - lib/fluent/plugin/systemd/pos_writer.rb
 homepage: https://github.com/reevoo/fluent-plugin-systemd
 licenses:
@@ -131,7 +133,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.6.9
+rubygems_version: 2.6.11
 signing_key: 
 specification_version: 4
 summary: Input plugin to read from systemd journal.