logstash-filter-kafka_time_machine 0.6.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4f04f3554198113091a6a21bdb645148cc4ac64d7a0189c26e749c33c9fc7262
-  data.tar.gz: 53d3bcf5901fd5cad1c8e2d5dc866399957eb90b284d9b41101cad050ebbb36e
+  metadata.gz: f54f2603dd73f8dd42e4677f51fdb40942cdb8ac918dcca99bcec2f022e16db2
+  data.tar.gz: cd3ec446acde4a0c82890ea4d38416ec62ab4081fa6f301be1a12a40ca287ae5
 SHA512:
-  metadata.gz: c6d5f460db066325cf1948f7da859950ce682f205daf429f05723e809f5393c89f6142f7ea3d1cf349c0c0840ab0ebe12e8699361187dc4e3368298a05510bab
-  data.tar.gz: 88f1bb5966c64b5c5b0f1399848d810db7150e4cf2855da894677a501587ee8f270c45aefb96d8a1dd3aee01ffa0e5bb9552bcda64cc3af16290820fb56c2723
+  metadata.gz: d65b9f86abd856b62dcb214eeba4a67074bfbb9bd67bdc2abdf3f99344df41bd788f029c7943d3d268c7eb88d05b5d7e5d606c86f0484a1558a8970a1299b07c
+  data.tar.gz: bb9dee15c942388e2261f85390f4ea0635723e5e6122162a5ebfa857676de0e312e683ba666d01326556310028b4fc0644e48e26e16fcaca79d092dbc542356b

data/README.md

@@ -1,3 +1,258 @@
-# logstash-filter-kafka_time_machine
+[![Gem Version](https://badge.fury.io/rb/logstash-filter-kafka_time_machine.svg)](https://badge.fury.io/rb/logstash-filter-kafka_time_machine)
 
-TBD
+# Logstash Plugin: logstash-filter-kafka_time_machine
+
+This is a filter plugin for [Logstash](https://github.com/elastic/logstash)
+
+## Description
+
+This filter plugin add additional fields to an existing event.  These fields provide additional metadata for tracking log events that have traversed multiple kafka and logstash blocks for aggregation.
+
+The typical flow for log events:
+
+```
+
+Service Log ---> kafka_shipper <--- logstash_shipper ---> | ott_network_link | ---> kafka_indexer <--- logstash_indexer ---> elastic_search
+
+```
+
+This filter leverages metadata inserted into the log event on both `logstash_shipper` and `logstash_indexer` nodes to track dwell time of log events through this pipeline.  When used the filter will add the following fields to the log event:
+
+| Event Field                         | Output Type | Description                                                                               |
+| ----------------------------------- | ----------- | ----------------------------------------------------------------------------------------- |
+| [ktm][datacenter_shipper]           | string      | Echo of `kafka_datacenter_shipper                                                         |
+| [ktm][kafka_topic_shipper]          | string      | Echo of `kafka_topic_shipper                                                              |
+| [ktm][kafka_consumer_group_shipper] | string      | Echo of `kafka_consumer_group_shipper                                                     |
+| [ktm][kafka_topic_indexer]          | string      | Echo of `kafka_topic_indexer                                                              |
+| [ktm][kafka_consumer_group_indexer] | string      | Echo of `kafka_consumer_group_indexer                                                     |
+| [ktm][payload_size_bytes]           | number      | If present in event, `payload` field size in bytes, else it's omitted                     |
+| [ktm][lag_shipper_ms]               | number      | Value of "`logstash_kafka_read_time_shipper - kafka_append_time_shipper`" in milliseconds |
+| [ktm][lag_indexer_ms]               | number      | Value of "`logstash_kafka_read_time_indexer - kafka_append_time_indexer`" in milliseconds |
+| [ktm][lag_total_ms]                 | number      | Value of "`logstash_kafka_read_time_indexer - kafka_append_time_shipper`" in milliseconds |
+
+What you do with the `[ktm]` event on it's return is use case specific. One currently used approach is to index this data in the ES document for log event it was generated against.  This provides metrics on a per document level in ES.
+
+## Kafka Time Machine Configuration Options
+
+This plugin supports the following configuration options:
+
+| Setting                                                               | Input Type | Required |
+| --------------------------------------------------------------------- | ---------- | -------- |
+| [kafka_datacenter_shipper](#kafka_datacenter_shipper)                 | string     | Yes      |
+| [kafka_topic_shipper](#kafka_topic_shipper)                           | string     | Yes      |
+| [kafka_consumer_group_shipper](#kafka_consumer_group_shipper)         | string     | Yes      |
+| [kafka_append_time_shipper](#kafka_append_time_shipper)               | string     | Yes      |
+| [logstash_kafka_read_time_shipper](#logstash_kafka_read_time_shipper) | string     | Yes      |
+| [kafka_topic_indexer](#kafka_topic_indexer)                           | string     | Yes      |
+| [kafka_consumer_group_indexer](#kafka_consumer_group_indexer)         | string     | Yes      |
+| [kafka_append_time_indexer](#kafka_append_time_indexer)               | string     | Yes      |
+| [logstash_kafka_read_time_indexer](#logstash_kafka_read_time_indexer) | string     | Yes      |
+
+
+
+### kafka_datacenter_shipper
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide datacenter that log event originated from; datacenter kafka_shipper is in.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    kafka_datacenter_shipper => "static_field"
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    kafka_datacenter_shipper => "%{[dynamic_field]}"
+  }
+}
+```
+
+### kafka_topic_shipper
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide kafka topic log event was read from on shipper.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    kafka_topic_shipper => "static_field"
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    kafka_topic_shipper => "%{[dynamic_field]}"
+  }
+}
+```
+
+### kafka_consumer_group_shipper
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide kafka consumer group log event was read from on shipper.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    kafka_consumer_group_shipper => "static_field"
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    kafka_consumer_group_shipper => "%{[dynamic_field]}"
+  }
+}
+```
+
+### kafka_append_time_shipper
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide EPOCH time in milliseconds log event was added to `kafka_shipper`.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    kafka_append_time_shipper => 1624394191000
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    kafka_append_time_shipper => "%{[dynamic_field]}"
+  }
+}
+```
+
+### logstash_kafka_read_time_shipper
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide EPOCH time in milliseconds log event read from to `kafka_shipper`.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    logstash_kafka_read_time_shipper => 1624394191000
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    logstash_kafka_read_time_shipper => "%{[dynamic_field]}"
+  }
+}
+```
+
+### kafka_topic_indexer
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide kafka topic log event was read from on indexer.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    kafka_topic_indexer => "static_field"
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    kafka_topic_indexer => "%{[dynamic_field]}"
+  }
+}
+```
+
+### kafka_consumer_group_indexer
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide kafka consumer group log event was read from on indexer.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    kafka_consumer_group_indexer => "static_field"
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    kafka_consumer_group_indexer => "%{[dynamic_field]}"
+  }
+}
+```
+
+### kafka_append_time_indexer
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide EPOCH time in milliseconds log event was added to `kafka_indexer`.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    kafka_append_time_indexer => 1624394191000
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    kafka_append_time_indexer => "%{[dynamic_field]}"
+  }
+}
+```
+
+### logstash_kafka_read_time_indexer
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide EPOCH time in milliseconds log event read from to `kafka_indexer`.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    logstash_kafka_read_time_indexer => 1624394191000
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    logstash_kafka_read_time_indexer => "%{[dynamic_field]}"
+  }
+}
+```

data/lib/logstash/filters/{kafkatimemachine.rb → kafka_time_machine.rb}

@@ -5,7 +5,34 @@ require "logstash/event"
 
 class LogStash::Filters::KafkaTimeMachine < LogStash::Filters::Base
 
-  config_name "kafkatimemachine"
+  config_name "kafka_time_machine"
+
+  # Datacenter the kafka message originated from.
+  config :kafka_datacenter_shipper, :validate => :string, :required => true
+
+  # Kafka Topic on shipper datacenter
+  config :kafka_topic_shipper, :validate => :string, :required => true
+
+  # Kafka Consumer Group on shipper datacenter
+  config :kafka_consumer_group_shipper, :validate => :string, :required => true
+
+  # Time message was appended to kafka on shipper datacenter
+  config :kafka_append_time_shipper, :validate => :string, :required => true
+
+  # Time message read from kafka by logstash on shipper datacenter
+  config :logstash_kafka_read_time_shipper, :validate => :string, :required => true
+
+  # Kafka Topic on indexer datacenter
+  config :kafka_topic_indexer, :validate => :string, :required => true
+
+  # Kafka Consumer Group on indexer datacenter
+  config :kafka_consumer_group_indexer, :validate => :string, :required => true
+
+  # Time message was appended to kafka on indexer datacenter
+  config :kafka_append_time_indexer, :validate => :string, :required => true
+
+  # Time message read from kafka by logstash on indexer datacenter
+  config :logstash_kafka_read_time_indexer, :validate => :string, :required => true
 
   public
   def register
@@ -16,11 +43,11 @@ class LogStash::Filters::KafkaTimeMachine < LogStash::Filters::Base
   def filter(event)
 
     # Extract shipper data and check for validity; note that kafka_datacenter_shipper is used for both shipper and indexer arrays
-    kafka_datacenter_shipper = event.get("[@metadata][kafka_datacenter_shipper]")
-    kafka_topic_shipper = event.get("[@metadata][kafka_topic_shipper]")
-    kafka_consumer_group_shipper = event.get("[@metadata][kafka_consumer_group_shipper]")
-    kafka_append_time_shipper = Float(event.get("[@metadata][kafka_append_time_shipper]")) rescue nil 
-    logstash_kafka_read_time_shipper = Float(event.get("[@metadata][logstash_kafka_read_time_shipper]")) rescue nil
+    kafka_datacenter_shipper         = event.sprintf(@kafka_datacenter_shipper)
+    kafka_topic_shipper              = event.sprintf(@kafka_topic_shipper)
+    kafka_consumer_group_shipper     = event.sprintf(@kafka_consumer_group_shipper)
+    kafka_append_time_shipper        = Float(event.sprintf(@kafka_append_time_shipper)) rescue nil 
+    logstash_kafka_read_time_shipper = Float(event.sprintf(@logstash_kafka_read_time_shipper)) rescue nil
 
     kafka_shipper_array = Array[kafka_datacenter_shipper, kafka_topic_shipper, kafka_consumer_group_shipper, kafka_append_time_shipper, logstash_kafka_read_time_shipper]
     @logger.debug("kafka_shipper_array: #{kafka_shipper_array}")
@@ -38,10 +65,10 @@ class LogStash::Filters::KafkaTimeMachine < LogStash::Filters::Base
     end
 
     # Extract indexer data and check for validity
-    kafka_topic_indexer = event.get("[@metadata][kafka_topic_indexer]")
-    kafka_consumer_group_indexer = event.get("[@metadata][kafka_consumer_group_indexer]")
-    kafka_append_time_indexer = Float(event.get("[@metadata][kafka_append_time_indexer]")) rescue nil
-    logstash_kafka_read_time_indexer = Float(event.get("[@metadata][logstash_kafka_read_time_indexer]")) rescue nil
+    kafka_topic_indexer              = event.sprintf(@kafka_topic_indexer)
+    kafka_consumer_group_indexer     = event.sprintf(@kafka_consumer_group_indexer)
+    kafka_append_time_indexer        = Float(event.sprintf(@kafka_append_time_indexer)) rescue nil
+    logstash_kafka_read_time_indexer = Float(event.sprintf(@logstash_kafka_read_time_indexer)) rescue nil
 
     kafka_indexer_array = Array[kafka_datacenter_shipper, kafka_topic_indexer, kafka_consumer_group_indexer, kafka_append_time_indexer, logstash_kafka_read_time_indexer]
     @logger.debug("kafka_indexer_array: #{kafka_indexer_array}")
@@ -66,7 +93,7 @@ class LogStash::Filters::KafkaTimeMachine < LogStash::Filters::Base
     elsif (indexer_valid == true && shipper_valid == false)
       event.set("[ktm]", {"lag_indexer_ms" => kafka_indexer_lag_ms, "datacenter_shipper" => kafka_datacenter_shipper, "kafka_topic_indexer" => kafka_topic_indexer, "kafka_consumer_group_indexer" => kafka_consumer_group_indexer, "tags" => ["ktm_lag_indexer"] })
     elsif (indexer_valid == false && shipper_valid == false)
-      @logger.debug("Error kafkatimemachine: Could not build valid response --> #{error_string_shipper}, #{error_string_indexer}")
+      @logger.debug("Error kafka_time_machine: Could not build valid response --> #{error_string_shipper}, #{error_string_indexer}")
     end
 
     # Add in the size of the payload field

data/logstash-filter-kafka_time_machine.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
     s.name = 'logstash-filter-kafka_time_machine'
-    s.version = '0.6.0'
+    s.version = '1.0.0'
     s.licenses = ['Apache-2.0']
     s.summary = "Calculate total time of logstash event that traversed 2 Kafka queues from a shipper site to an indexer site"
     s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-filter-kafka_time_machine
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 1.0.0
 platform: ruby
 authors:
 - Chris Foster
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2021-06-21 00:00:00.000000000 Z
+date: 2021-06-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: logstash-core-plugin-api
@@ -54,7 +54,7 @@ extra_rdoc_files: []
 files:
 - Gemfile
 - README.md
-- lib/logstash/filters/kafkatimemachine.rb
+- lib/logstash/filters/kafka_time_machine.rb
 - logstash-filter-kafka_time_machine.gemspec
 homepage: http://www.elastic.co/guide/en/logstash/current/index.html
 licenses: