logstash-filter-kafka_time_machine 0.5.0 → 2.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f737fcbdc55c777f9ff5000e778c8fd26ceda00fdb87ee48ecb6c65d7dafaf90
-  data.tar.gz: 5c4f1cd0dc80f63cd97a93701c2d12d79e0c97a93f913187c6dcaa1e19ca306a
+  metadata.gz: dcb429711e99220eb57f095154d68f2b3f477956587762c4ba03ef7bc434ba30
+  data.tar.gz: 3c66923274a218187bef908747091baa540523729c6d961f02ab002eb6701fce
 SHA512:
-  metadata.gz: e0549b7084ce70c4af5e94d249ebf244d215f4bffbdc4db5c9b0356041cc50b736e0e2dcf0680b0d049b0f38402a9f3931aa2eae6ace9b41cd3628ae04c73fe0
-  data.tar.gz: '04338e54be18ff407221ed442d62d641cbcf4391adb5cd85d919efa0c37fe158f18bcfa88cace01c69f8ed44b7156a1ac96e9e6b42124fed87a1873d4e49eee1'
+  metadata.gz: 840b0fbdef1e7096c2e51cdaac3a4ef38e5e7830fb8263e8511d8e09116a13a92a90dcf13ffb72ffef51da4f66aaf898235d30ee50c6f7adddab3c79a428bb80
+  data.tar.gz: 6ffbb0731c74f2b7168dccd79f4a78f00b168baab2d5d111ac9a872b39c24f3aca8fa5b92e67053a7f5abef71a0fc44561604f56f669c40812a10c237e5b65e9

data/README.md

@@ -1,3 +1,374 @@
-# logstash-filter-kafka_time_machine
+[![Gem Version](https://badge.fury.io/rb/logstash-filter-kafka_time_machine.svg)](https://badge.fury.io/rb/logstash-filter-kafka_time_machine)
 
-TBD
+# Logstash Plugin: logstash-filter-kafka_time_machine
+
+This is a filter plugin for [Logstash](https://github.com/elastic/logstash)
+
+## Description
+
+This filter plugin will generate new events in the logstash pipeline.  These events are generated based on fields in the original that are extracted and passed to the filter.  The new events generate metrics for log events that have traversed multiple kafka and logstash blocks for aggregation.
+
+The typical flow for log events:
+
+```
+
+Service Log ---> kafka_shipper <--- logstash_shipper ---> | ott_network_link | ---> kafka_indexer <--- logstash_indexer ---> elastic_search
+
+```
+
+The filter leverages metadata inserted into the log event on both `logstash_shipper` and `logstash_indexer` nodes to track dwell time of log events through this pipeline.
+
+## Kafka Time Machine Result
+
+When the `kafka_time_machine` executes it will return a [InfluxDB Line Protocol](https://docs.influxdata.com/influxdb/v1.8/write_protocols/line_protocol_tutorial/) formatted metric, i.e.:
+
+```
+ktm,datacenter=kafka_datacenter_shipper-test,lag_type=total,owner=ktm_test@cisco.com lag_ms=300i,payload_size_bytes=40i 1634662795000000000
+```
+
+The plugin will also emit a metric if an error was encountered, i.e.:
+
+```
+ktm_error,datacenter=kafka_datacenter_shipper-test,owner=ktm_test@cisco.com,source=shipper count=1i 1634662795000000000
+```
+
+To ensure a logstash `output{}` block can properly route this metric, the new event are tagged with a `[@metadata][ktm_tag][ktm_metric]` field, i.e.:
+
+```
+{
+    "ktm_metric" => "ktm,datacenter=kafka_datacenter_shipper-test,lag_type=total,owner=ktm_test@cisco.com lag_ms=300i,payload_size_bytes=40i 1634662795000000000",
+    "@timestamp" => 2021-10-20T23:46:24.704Z,
+     "@metadata" => {
+        "ktm_tags" => {
+            "ktm_metric" => "true"
+        }
+    },
+      "@version" => "1"
+}
+```
+
+### Metric Event Breakdown
+
+The `kafka_time_machine` can insert one or more new events in the pipeline.  The `ktm_metric` created will be one of:
+
+- `ktm`
+- `ktm_error`
+
+In the case of `ktm` the metric breakdown is:
+
+| Line Protocol Element | Line Protocol Type | Description                                 |
+| --------------------- | ------------------ | ------------------------------------------- |
+| datacenter            | tag                | Echo of `kafka_datacenter_shipper`          |
+| lag_type              | tag                | Calculated lag type                         |
+| owner                 | tag                | Echo of `event_owner`                       |
+| lag_ms                | field              | Calculated lag in milliseconds              |
+| payload_size_bytes    | field              | Calculated size of `payload` field in bytes |
+
+Meaning of `lag_type`:
+
+- `total`: Lag calculated includes dwell time on both on shipper and indexer
+- `indexer`: Lag calculated is dwell time for indexer only.  Insufficient data provided for shipper to compute `total` lag.
+- `shipper`: Lag calculated is dwell time for shipper only.  Insufficient data provided for indexer to compute `total` lag.
+
+In the case of `ktm_error` the metric breakdown is:
+
+| Line Protocol Element | Line Protocol Type | Description                          |
+| --------------------- | ------------------ | ------------------------------------ |
+| datacenter            | tag                | Echo of `kafka_datacenter_shipper`   |
+| source                | tag                | Source of the error metric           |
+| owner                 | tag                | Echo of `event_owner`                |
+| count                 | field              | Count to track error; not cumulative |
+
+Meaning of `source`:
+
+- `indexer`: Insufficient data provided for indexer to compute `total` lag.
+- `shipper`: Insufficient data provided for shipper to compute `total` lag.
+- `insufficient_data`: Insufficient data provided both indexer and shipper to compute `total` lag.
+- `unknown`: Unknown error encountered
+
+### Metric Event Timestamp
+
+When the `kafka_time_machine` generates the [InfluxDB Line Protocol](https://docs.influxdata.com/influxdb/v1.8/write_protocols/line_protocol_tutorial/) metric it must also set the timestamp on the event.  To ensure the caller of filter has control of this the `event_time_ms` configuration is used to set the metric timestamp.
+
+For example if `event_time_ms` is provided as `1634662795000` the resulting metric would be:
+
+```
+ktm,datacenter=kafka_datacenter_shipper-test,lag_type=total,owner=ktm_test@cisco.com lag_ms=300i,payload_size_bytes=40i 1634662795000000000
+```
+
+## Kafka Time Machine Configuration Options
+
+This plugin requires the following configurations:
+
+| Setting                                                               | Input Type | Required |
+| --------------------------------------------------------------------- | ---------- | -------- |
+| [kafka_datacenter_shipper](#kafka_datacenter_shipper)                 | string     | Yes      |
+| [kafka_topic_shipper](#kafka_topic_shipper)                           | string     | Yes      |
+| [kafka_consumer_group_shipper](#kafka_consumer_group_shipper)         | string     | Yes      |
+| [kafka_append_time_shipper](#kafka_append_time_shipper)               | string     | Yes      |
+| [logstash_kafka_read_time_shipper](#logstash_kafka_read_time_shipper) | string     | Yes      |
+| [kafka_topic_indexer](#kafka_topic_indexer)                           | string     | Yes      |
+| [kafka_consumer_group_indexer](#kafka_consumer_group_indexer)         | string     | Yes      |
+| [kafka_append_time_indexer](#kafka_append_time_indexer)               | string     | Yes      |
+| [logstash_kafka_read_time_indexer](#logstash_kafka_read_time_indexer) | string     | Yes      |
+| [event_owner](#event_owner)                                           | string     | Yes      |
+| [event_time_ms](#event_time_ms)                                       | string     | Yes      |
+
+> Why are all settings required?
+>
+>> This was a design decision based on the use case.  Tracking a Kafka "lag by time" metric, but not knowing the topic and consumer group would be essentially useless.  By leveraging the [Kafka input `decorate_events`](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#_metadata_fields) feature we know we'll always have the required fields.  
+>>
+>> While they are required, they can be passed as empty strings.  The plugin will handle these cases, i.e. the `kafka_consumer_group_shipper` name is empty string, and only return `indexer` results
+
+### kafka_datacenter_shipper
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide datacenter that log event originated from; datacenter kafka_shipper is in.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    kafka_datacenter_shipper => "static_field"
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    kafka_datacenter_shipper => "%{[dynamic_field]}"
+  }
+}
+```
+
+### kafka_topic_shipper
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide kafka topic log event was read from on shipper.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    kafka_topic_shipper => "static_field"
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    kafka_topic_shipper => "%{[dynamic_field]}"
+  }
+}
+```
+
+### kafka_consumer_group_shipper
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide kafka consumer group log event was read from on shipper.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    kafka_consumer_group_shipper => "static_field"
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    kafka_consumer_group_shipper => "%{[dynamic_field]}"
+  }
+}
+```
+
+### kafka_append_time_shipper
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide EPOCH time in milliseconds log event was added to `kafka_shipper`.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    kafka_append_time_shipper => 1624394191000
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    kafka_append_time_shipper => "%{[dynamic_field]}"
+  }
+}
+```
+
+### logstash_kafka_read_time_shipper
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide EPOCH time in milliseconds log event read from to `kafka_shipper`.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    logstash_kafka_read_time_shipper => 1624394191000
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    logstash_kafka_read_time_shipper => "%{[dynamic_field]}"
+  }
+}
+```
+
+### kafka_topic_indexer
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide kafka topic log event was read from on indexer.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    kafka_topic_indexer => "static_field"
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    kafka_topic_indexer => "%{[dynamic_field]}"
+  }
+}
+```
+
+### kafka_consumer_group_indexer
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide kafka consumer group log event was read from on indexer.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    kafka_consumer_group_indexer => "static_field"
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    kafka_consumer_group_indexer => "%{[dynamic_field]}"
+  }
+}
+```
+
+### kafka_append_time_indexer
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide EPOCH time in milliseconds log event was added to `kafka_indexer`.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    kafka_append_time_indexer => 1624394191000
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    kafka_append_time_indexer => "%{[dynamic_field]}"
+  }
+}
+```
+
+### logstash_kafka_read_time_indexer
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide EPOCH time in milliseconds log event read from to `kafka_indexer`.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    logstash_kafka_read_time_indexer => 1624394191000
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    logstash_kafka_read_time_indexer => "%{[dynamic_field]}"
+  }
+}
+```
+
+### event_owner
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide event owner; represents the owner of the log.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    event_owner => "static_field"
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    event_owner => "%{[dynamic_field]}"
+  }
+}
+```
+
+### event_time_ms
+
+- Value type is [string](https://www.elastic.co/guide/en/logstash/7.13/configuration-file-structure.html#string)
+- There is no default value for this setting.
+
+Provide EPOCH time in milliseconds that this event is being processed.  This time will be appending the generated InfluxDb Line Protocol metric.  Field values can be static or dynamic:
+
+```
+filter {
+  kafka_time_machine {
+    event_time_ms => 1624394191000
+  }
+}
+```
+
+```
+filter {
+  kafka_time_machine {
+    event_time_ms => "%{[dynamic_field]}"
+  }
+}
+```

data/lib/logstash/filters/kafka_time_machine.rb

@@ -0,0 +1,252 @@
+# encoding: utf-8
+require "logstash/filters/base"
+require "logstash/namespace"
+require "logstash/event"
+require "influxdb-client"
+
+class LogStash::Filters::KafkaTimeMachine < LogStash::Filters::Base
+
+  config_name "kafka_time_machine"
+
+  # Datacenter the kafka message originated from.
+  config :kafka_datacenter_shipper, :validate => :string, :required => true
+
+  # Kafka Topic on shipper datacenter
+  config :kafka_topic_shipper, :validate => :string, :required => true
+
+  # Kafka Consumer Group on shipper datacenter
+  config :kafka_consumer_group_shipper, :validate => :string, :required => true
+
+  # Time message was appended to kafka on shipper datacenter
+  config :kafka_append_time_shipper, :validate => :string, :required => true
+
+  # Time message read from kafka by logstash on shipper datacenter
+  config :logstash_kafka_read_time_shipper, :validate => :string, :required => true
+
+  # Kafka Topic on indexer datacenter
+  config :kafka_topic_indexer, :validate => :string, :required => true
+
+  # Kafka Consumer Group on indexer datacenter
+  config :kafka_consumer_group_indexer, :validate => :string, :required => true
+
+  # Time message was appended to kafka on indexer datacenter
+  config :kafka_append_time_indexer, :validate => :string, :required => true
+
+  # Time message read from kafka by logstash on indexer datacenter
+  config :logstash_kafka_read_time_indexer, :validate => :string, :required => true
+
+  # Owner of the event currenty being process.
+  config :event_owner, :validate => :string, :required => true
+
+  # Current time since EPOCH in ms that should be set in the influxdb generated metric
+  config :event_time_ms, :validate => :string, :required => true
+
+  # Current time since EPOCH in ms that should be set in the influxdb generated metric
+  config :elasticsearch_cluster, :validate => :string, :required => true
+
+  # Current time since EPOCH in ms that should be set in the influxdb generated metric
+  config :elasticsearch_cluster_index, :validate => :string, :required => true
+
+  public
+  def register
+  
+  end
+
+  public
+  def filter(event)
+
+    @logger.debug("Starting filter calculations")
+
+    # Note - It was considered to error check for strings that are invalid, i.e. "%{[@metadata][ktm][kafka_datacenter_shipper]}".  However, this string being present is a good way to identify 
+    # shipper/indexer logstash configs that are wrong so its allowed to pass through unaltered.
+    #
+    # Extract all string values to local variables.
+    event_owner                      = event.sprintf(@event_owner)
+    shipper_kafka_datacenter         = event.sprintf(@kafka_datacenter_shipper)
+    shipper_kafka_topic              = event.sprintf(@kafka_topic_shipper)
+    shipper_kafka_consumer_group     = event.sprintf(@kafka_consumer_group_shipper)
+    indexer_kafka_topic              = event.sprintf(@kafka_topic_indexer)
+    indexer_kafka_consumer_group     = event.sprintf(@kafka_consumer_group_indexer)
+    elasticsearch_cluster            = event.sprintf(@elasticsearch_cluster)
+    elasticsearch_cluster_index      = event.sprintf(@elasticsearch_cluster_index)
+
+    # Extract all the "time" related values to local variables.  This need special handling due to the Float() operation.
+    #
+    # We must check for a valid numberic value; if not the float operation will error out on "invalid hash error" and stop logstash pipeline
+    event_time_ms                    = get_numeric(event.sprintf(@event_time_ms))
+    shipper_kafka_append_time        = get_numeric(event.sprintf(@kafka_append_time_shipper))
+    shipper_logstash_kafka_read_time = get_numeric(event.sprintf(@logstash_kafka_read_time_shipper))
+    indexer_kafka_append_time        = get_numeric(event.sprintf(@kafka_append_time_indexer))
+    indexer_logstash_kafka_read_time = get_numeric(event.sprintf(@logstash_kafka_read_time_indexer))
+    
+    # Validate the shipper data
+    shipper_kafka_array = Array[shipper_kafka_datacenter, shipper_kafka_topic, shipper_kafka_consumer_group, shipper_kafka_append_time, shipper_logstash_kafka_read_time, event_owner, event_time_ms, elasticsearch_cluster, elasticsearch_cluster_index]
+    if (shipper_kafka_array.any? { |text| text.nil? || text.to_s.empty? })
+      @logger.debug("shipper_kafka_array invalid: Found null")
+      error_string_shipper = sprintf("Error in shipper data: %s", shipper_kafka_array)
+      @logger.debug(error_string_shipper)
+      shipper_valid = false
+    else
+      @logger.debug("shipper_kafka_array valid")
+      shipper_valid = true
+      shipper_logstash_kafka_read_time = shipper_logstash_kafka_read_time.to_i
+      shipper_kafka_append_time        = shipper_kafka_append_time.to_i
+      shipper_kafka_lag_ms             = shipper_logstash_kafka_read_time - shipper_kafka_append_time
+    end
+
+    # Validate the indexer data
+    indexer_kafka_array = Array[shipper_kafka_datacenter, indexer_kafka_topic, indexer_kafka_consumer_group, indexer_kafka_append_time, indexer_logstash_kafka_read_time, event_owner, event_time_ms, elasticsearch_cluster, elasticsearch_cluster_index]
+    if (indexer_kafka_array.any? { |text| text.nil? || text.to_s.empty? })
+      @logger.debug("indexer_kafka_array invalid: Found null")
+      error_string_indexer = sprintf("Error in indexer data: %s", indexer_kafka_array)
+      @logger.debug(error_string_indexer)
+      indexer_valid = false
+    else
+      @logger.debug("indexer_kafka_array valid")
+      indexer_valid = true
+      indexer_logstash_kafka_read_time = indexer_logstash_kafka_read_time.to_i
+      indexer_kafka_append_time        = indexer_kafka_append_time.to_i
+      indexer_kafka_lag_ms             = indexer_logstash_kafka_read_time - indexer_kafka_append_time
+    end
+
+    # Add in the size of the payload field
+    payload_bytesize = 0
+    if event.get("[payload]")
+      payload_bytesize = event.get("[payload]").bytesize
+    end
+
+    # Set time (nanoseconds) for influxdb line protocol
+    epoch_time_ns = nil
+    if (event_time_ms != nil )
+      epoch_time_ns = event_time_ms * 1000000
+    end
+
+    # Create array to hold one or more ktm metric events
+    ktm_metric_event_array = Array.new
+
+    # Populate the event and set tags
+    if (shipper_valid == true && indexer_valid == true && epoch_time_ns != nil)
+      total_kafka_lag_ms = indexer_logstash_kafka_read_time - shipper_kafka_append_time
+
+      point_influxdb = create_influxdb_point_ktm(shipper_kafka_datacenter, event_owner, payload_bytesize, "total", total_kafka_lag_ms, epoch_time_ns, elasticsearch_cluster, elasticsearch_cluster_index)
+      ktm_metric_event_array.push point_influxdb
+
+    elsif (shipper_valid == true && indexer_valid == false && epoch_time_ns != nil)
+      point_influxdb = create_influxdb_point_ktm(shipper_kafka_datacenter, event_owner, payload_bytesize, "shipper", shipper_kafka_lag_ms, epoch_time_ns, elasticsearch_cluster, elasticsearch_cluster_index)
+      ktm_metric_event_array.push point_influxdb
+
+      point_influxdb = create_influxdb_point_ktm_error(shipper_kafka_datacenter, event_owner, epoch_time_ns, "indexer", elasticsearch_cluster, elasticsearch_cluster_index)
+      ktm_metric_event_array.push point_influxdb
+
+    elsif (indexer_valid == true && shipper_valid == false && epoch_time_ns != nil)
+      point_influxdb = create_influxdb_point_ktm(shipper_kafka_datacenter, event_owner, payload_bytesize, "indexer", indexer_kafka_lag_ms, epoch_time_ns, elasticsearch_cluster, elasticsearch_cluster_index)
+      ktm_metric_event_array.push point_influxdb
+
+      point_influxdb = create_influxdb_point_ktm_error(shipper_kafka_datacenter, event_owner, epoch_time_ns, "shipper", elasticsearch_cluster, elasticsearch_cluster_index)
+      ktm_metric_event_array.push point_influxdb
+
+    elsif (indexer_valid == false && shipper_valid == false)
+
+      point_influxdb = create_influxdb_point_ktm_error(shipper_kafka_datacenter, event_owner, epoch_time_ns, "insufficient_data", elasticsearch_cluster, elasticsearch_cluster_index)
+      ktm_metric_event_array.push point_influxdb
+
+      error_string = sprintf("Error kafka_time_machine: Could not build valid response --> %s, %s", error_string_shipper, error_string_indexer)
+      @logger.debug(error_string)
+    
+    else
+
+      point_influxdb = create_influxdb_point_ktm_error(shipper_kafka_datacenter, event_owner, epoch_time_ns, "unknown", elasticsearch_cluster, elasticsearch_cluster_index)
+      ktm_metric_event_array.push point_influxdb
+
+      error_string = "Unknown error encountered"
+      @logger.debug(error_string)
+
+    end
+
+    # Publish even event in our array
+    ktm_metric_event_array.each do |metric_event|
+      
+      # Create new event for KTM metric
+      event_ktm = LogStash::Event.new
+
+      event_ktm.set("ktm_metric", metric_event)
+      event_ktm.set("[@metadata][ktm_tags][ktm_metric]", "true")
+
+      filter_matched(event_ktm)
+      yield event_ktm
+    
+    end
+
+  end # def filter
+
+  # Creates an Influx DB line-protocol data point to return
+  public
+  def create_influxdb_point_ktm(datacenter, event_owner, payload_size_bytes, lag_type, lag_ms, epoch_time_ns, elasticsearch_cluster, elasticsearch_cluster_index)
+
+    point = InfluxDB2::Point.new( name: "ktm",
+                                  tags: {datacenter: datacenter, owner: event_owner, lag_type: lag_type, es_cluster: elasticsearch_cluster, es_cluster_index: elasticsearch_cluster_index},
+                                  fields: {payload_size_bytes: payload_size_bytes, lag_ms: lag_ms},
+                                  time: epoch_time_ns)
+
+    point_influxdb = point.to_line_protocol
+    return point_influxdb
+
+  end # def create_influxdb_point
+
+  # Creates an Influx DB line-protocol data point to return
+  public
+  def create_influxdb_point_ktm_error(datacenter, event_owner, epoch_time_ns, type, elasticsearch_cluster, elasticsearch_cluster_index)
+
+    # Check for nil values
+    if (nil == datacenter)
+      datacenter = "unknown"
+    end
+
+    if (nil == event_owner)
+      event_owner = "unknown"
+    end
+
+    # set time if we didn't recieve it
+    if (nil == epoch_time_ns)
+      epoch_time_ns = ((Time.now.to_f * 1000).to_i)*1000000
+    end
+
+    point = InfluxDB2::Point.new( name: "ktm_error",
+                                  tags: {datacenter: datacenter, owner: event_owner, source: type, es_cluster: elasticsearch_cluster, es_cluster_index: elasticsearch_cluster_index},
+                                  fields: {count: 1},
+                                  time: epoch_time_ns)
+
+    point_influxdb = point.to_line_protocol
+    return point_influxdb
+
+  end # def create_influxdb_point
+
+  # Ensures the provided value is numeric; if not returns 'nil'
+  public
+  def get_numeric(input_str) 
+
+    # @logger.debug("Aggregate timeout for '#{@task_id}' pattern: #{@timeout} seconds")
+    @logger.debug("get_numeric operating on: #{input_str} ")
+    
+    is_numeric = input_str.to_s.match(/\A[+-]?\d+?(\.\d+)?\Z/) == nil ? false : true
+    if (true == is_numeric)
+      @logger.debug("get_numeric - valid value provided")
+      num_value = Float(sprintf(input_str))
+      
+      if (false == num_value.positive?)
+        @logger.debug("get_numeric - negative value provided")
+        num_value = nil
+      end
+
+    else
+      @logger.debug("get_numeric - invalid value provided")
+      num_value = nil
+    end
+
+    @logger.debug(sprintf("get_numeric response --> #{num_value}"))
+
+    return num_value
+
+  end # def get_numberic
+
+end # class LogStash::Filters::KafkaTimeMachine

data/logstash-filter-kafka_time_machine.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
     s.name = 'logstash-filter-kafka_time_machine'
-    s.version = '0.5.0'
+    s.version = '2.0.1'
     s.licenses = ['Apache-2.0']
     s.summary = "Calculate total time of logstash event that traversed 2 Kafka queues from a shipper site to an indexer site"
     s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
@@ -20,5 +20,6 @@ Gem::Specification.new do |s|
 
     # Gem dependencies
     s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
+    s.add_runtime_dependency "influxdb-client", "~> 2.0.0"
     s.add_development_dependency 'logstash-devutils', '~> 0'
   end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-filter-kafka_time_machine
 version: !ruby/object:Gem::Version
-  version: 0.5.0
+  version: 2.0.1
 platform: ruby
 authors:
 - Chris Foster
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2021-06-17 00:00:00.000000000 Z
+date: 2021-10-28 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: logstash-core-plugin-api
@@ -30,6 +30,20 @@ dependencies:
     - - "<="
       - !ruby/object:Gem::Version
         version: '2.99'
+- !ruby/object:Gem::Dependency
+  name: influxdb-client
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 2.0.0
 - !ruby/object:Gem::Dependency
   name: logstash-devutils
   requirement: !ruby/object:Gem::Requirement
@@ -54,7 +68,7 @@ extra_rdoc_files: []
 files:
 - Gemfile
 - README.md
-- lib/logstash/filters/kafkatimemachine.rb
+- lib/logstash/filters/kafka_time_machine.rb
 - logstash-filter-kafka_time_machine.gemspec
 homepage: http://www.elastic.co/guide/en/logstash/current/index.html
 licenses:

data/lib/logstash/filters/kafkatimemachine.rb

@@ -1,83 +0,0 @@
-# encoding: utf-8
-require "logstash/filters/base"
-require "logstash/namespace"
-require "logstash/event"
-
-class LogStash::Filters::KafkaTimeMachine < LogStash::Filters::Base
-
-  config_name "kafkatimemachine"
-
-  public
-  def register
-  
-  end
-
-  public
-  def filter(event)
-
-    # Extract shipper data and check for validity; note that kafka_datacenter_shipper is used for both shipper and indexer arrays
-    kafka_datacenter_shipper = event.get("[@metadata][kafka_datacenter_shipper]")
-    kafka_topic_shipper = event.get("[@metadata][kafka_topic_shipper]")
-    kafka_consumer_group_shipper = event.get("[@metadata][kafka_consumer_group_shipper]")
-    kafka_append_time_shipper = Float(event.get("[@metadata][kafka_append_time_shipper]")) rescue nil 
-    logstash_kafka_read_time_shipper = Float(event.get("[@metadata][logstash_kafka_read_time_shipper]")) rescue nil
-
-    kafka_shipper_array = Array[kafka_datacenter_shipper, kafka_topic_shipper, kafka_consumer_group_shipper, kafka_append_time_shipper, logstash_kafka_read_time_shipper]
-    @logger.debug("kafka_shipper_array: #{kafka_shipper_array}")
-
-    if (kafka_shipper_array.any? { |text| text.nil? || text.to_s.empty? })
-      @logger.debug("kafka_shipper_array invalid: Found null")
-      error_string_shipper = "Error in shipper data: #{kafka_shipper_array}"
-      shipper_valid = false
-    else
-      @logger.debug("kafka_shipper_array valid")
-      shipper_valid = true
-      logstash_kafka_read_time_shipper = logstash_kafka_read_time_shipper.to_i
-      kafka_append_time_shipper = kafka_append_time_shipper.to_i
-      kafka_shipper_lag_ms = logstash_kafka_read_time_shipper - kafka_append_time_shipper
-    end
-
-    # Extract indexer data and check for validity
-    kafka_topic_indexer = event.get("[@metadata][kafka_topic_indexer]")
-    kafka_consumer_group_indexer = event.get("[@metadata][kafka_consumer_group_indexer]")
-    kafka_append_time_indexer = Float(event.get("[@metadata][kafka_append_time_indexer]")) rescue nil
-    logstash_kafka_read_time_indexer = Float(event.get("[@metadata][logstash_kafka_read_time_indexer]")) rescue nil
-
-    kafka_indexer_array = Array[kafka_datacenter_shipper, kafka_topic_indexer, kafka_consumer_group_indexer, kafka_append_time_indexer, logstash_kafka_read_time_indexer]
-    @logger.debug("kafka_indexer_array: #{kafka_indexer_array}")
-
-    if (kafka_indexer_array.any? { |text| text.nil? || text.to_s.empty? })
-      @logger.debug("kafka_indexer_array invalid: Found null")
-      error_string_indexer = "Error in indexer data: #{kafka_indexer_array}"
-      indexer_valid = false
-    else
-      @logger.debug("kafka_indexer_array valid")
-      indexer_valid = true
-      logstash_kafka_read_time_indexer = logstash_kafka_read_time_indexer.to_i
-      kafka_append_time_indexer = kafka_append_time_indexer.to_i
-      kafka_indexer_lag_ms = logstash_kafka_read_time_indexer - kafka_append_time_indexer
-    end
-
-    if (shipper_valid == true && indexer_valid == true)
-      kafka_total_lag_ms = logstash_kafka_read_time_indexer - kafka_append_time_shipper
-      event.set("[ktm]", {"lag_total_ms" => kafka_total_lag_ms, "lag_indexer" => kafka_indexer_lag_ms, "lag_shipper" => kafka_shipper_lag_ms, "datacenter_shipper" => kafka_datacenter_shipper, "kafka_topic_indexer" => kafka_topic_indexer, "kafka_consumer_group_indexer" => kafka_consumer_group_indexer, "kafka_topic_shipper" => kafka_topic_shipper, "kafka_consumer_group_shipper" => kafka_consumer_group_shipper, "tags" => ["ktm_lag_total"] })
-    elsif (shipper_valid == true && indexer_valid == false)
-      event.set("[ktm]", {"lag_shipper_ms" => kafka_shipper_lag_ms, "datacenter_shipper" => kafka_datacenter_shipper, "kafka_topic_shipper" => kafka_topic_shipper, "kafka_consumer_group_shipper" => kafka_consumer_group_shipper, "tags" => ["ktm_lag_shipper"] })
-    elsif (indexer_valid == true && shipper_valid == false)
-      event.set("[ktm]", {"lag_indexer_ms" => kafka_indexer_lag_ms, "datacenter_shipper" => kafka_datacenter_shipper, "kafka_topic_indexer" => kafka_topic_indexer, "kafka_consumer_group_indexer" => kafka_consumer_group_indexer, "tags" => ["ktm_lag_indexer"] })
-    elsif (indexer_valid == false && shipper_valid == false)
-      @logger.debug("Error kafkatimemachine: Could not build valid response --> #{error_string_shipper}, #{error_string_indexer}")
-    end
-
-    # Add in the size of the payload field
-    if event.get("[payload]")
-      payload_bytesize = event.get("[payload]").bytesize
-      event.set("[ktm][payload_size_bytes]", payload_bytesize)
-    end
-
-    # filter_matched should go in the last line of our successful code
-    filter_matched(event)
-
-  end # def filter
-
-end # class LogStash::Filters::KafkaTimeMachine