logstash-codec-mtrraw 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: d88e82a64b32f90104a3e416cec993d4a72335b4
+  data.tar.gz: 99542d614c33bedbe63b114c1163f7bdd2ae1d1f
+SHA512:
+  metadata.gz: 21b9f69b4dcc2dc982b2ea9b4fa30a684992776294210c5ed5bbb7f82a6484507dc05342db5e50cb960ab5db2c0347ab9c59973106e559e7959cb46e5f8b3cfc
+  data.tar.gz: cb1badb8fc189389e7a68552de80e970477107cbad5a3bec6635738b31a60f98cf52552ba356479dbccf0882ce563caada82b07305896a4b5efe0b6f9a0ef7f3

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+## 0.1.0
+  - Plugin created with the logstash plugin generator

data/CONTRIBUTORS

@@ -0,0 +1,10 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* svdasein - daveparker01@gmail.com
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+gemspec
+

data/LICENSE

@@ -0,0 +1,11 @@
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,88 @@
+# Logstash Plugin
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+The logstash mtrraw codec wraps together a bunch of logic that makes it easy to use mtr --raw output in your ELK infrastructure.  
+
+### Installation
+
+```
+bin/logstash-plugin install logstash-codec-mtrraw
+```
+### Configuration
+
+```
+input {
+        tcp {
+                port => 4327
+                codec => "mtrraw"
+        }
+}
+```
+
+### Sending MTR trace data
+Feed it with something that's functionally equivalent to this:
+
+```
+while true ; do (echo "s 0 GOOGDNS 1";mtr --raw --no-dns  -c 1 8.8.8.8 ) | awk '{printf $0";"}'  | nc localhost 4327 ; done
+```
+
+Put the above in a script, make the script executable, and run it in the background.  It'll continuously feed mtr trace data to
+the codec.
+
+Explanation:
+
+There's an infinite loop around the traces without a pause.  A pause isn't really needed to keep load down as the trace is i/o bound
+on the network all the time anyway.
+
+The `(echo ...;mtr)` construct allows us to overload the frontend of the trace a little bit and have the whole thing treated as a single
+stream.  The front of the trace has a line that looks like this:
+
+```
+s 0 <targetname> <pingcount>
+
+```
+
+* <targetname> is whatever name you want to give the trace
+* <pingcount> is the number of pings you're going to be doing to each node in the trace. This must match the -c parameter to mtr (see below).
+
+Modify the echo statement accordingly.
+
+The MTR execution part requires the following:
+
+* You must use the `--raw` output format
+* You must specify the `-c` (count) option to state how many pings you want to do. This number must also be in the start line (above)
+
+Any other options are optional :)
+
+The `| awk '{printf $0";"}` construct takes all the --raw output lines and puts them together in one line delimited by semicolons.  This
+in turn is sent to your logstash instance at the port you defined when you configured it using a tcp connection via the netcat (nc) command.
+
+### What you'll get
+
+The codec generates two kinds of documents:
+
+#### wholepath
+
+Whole path documents are identified by the "wholepath" tag.  These documents describe the entire path taken and assign a signature to the path you can 
+use to identify it among paths taken.  This allows you to e.g. do route flap detection, rtt & loss analysis, etc.
+
+#### hop
+
+Hop documents are identified by the "hop" tag.  These documents describe a single hop on a path (where the path is identified by the same identifier
+that the wholepath document contains).  You can use these with the excellent [Network visualization plugin](https://github.com/dlumbrer/kbn_network)  to visualize routes in kibana.
+
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/codecs/mtrraw.rb

@@ -0,0 +1,132 @@
+# encoding: utf-8
+require "logstash/codecs/base"
+require "logstash/codecs/line"
+require "logstash/namespace"
+require "securerandom"
+require "digest"
+
+
+# This codec presumes you've somehow sent in the equivalent of this
+# bash one-liner:
+# (echo "S MYTARGET" ; mtr --raw -c <samplecount> 8.8.8.8) | awk '{printf $1";"}'
+# You can get that into logstash any way you want, e.g. netcat will
+# work if you set up a tcp input:
+# (echo "S MYTARGET" ; mtr --raw -c <samplecount> 8.8.8.8) | awk '{printf $1";"}' | nc <myserver> <myport>
+#
+
+class MtrRec
+	attr_accessor :type,:id,:data
+	def initialize(line)
+		parts = line.split(/\s+/,3)
+		@type = parts[0]
+		@id = parts[1]
+		@data = parts[2]
+	end
+end
+
+class MtrHost
+	attr_accessor :hostid,:addr,:pings,:dns,:recs,:totalreplies
+	def initialize(rec,pingcount,recs)
+		@hostid = rec.id
+		@addr = rec.data
+		@recs = recs
+		@pings = recs.select{|each| each.type == 'p'}.collect {|each|each.data}
+		@pingloss = (100.to_f - (100.to_f * (@pings.size.to_f / pingcount.to_f))).to_i if pingcount.to_i > 0
+		@avgrtt = (@pings.inject(0.0) {|counter,each| counter += (each.to_f/1000)} / @pings.size).to_i
+		@dns = recs.select{|each| each.type =='d'}.collect {|each|each.data}.pop
+	end
+	def to_event_struct
+		{:hostid => @hostid, :addr => @addr , :pings => @pings ,:dns => @dns,:pingloss => @pingloss,:avgrtt => @avgrtt}
+	end
+end 
+
+class LogStash::Codecs::Mtrraw < LogStash::Codecs::Base
+
+  # The codec name
+  config_name "mtrraw"
+
+  # Append a string to the message
+  # config :append, :validate => :string, :default => ', Hello World!'
+
+  def register
+  end # def register
+
+  def decode(data)
+    mtrlines = data.split(';')
+    mtrrecs = mtrlines.collect {|each| MtrRec.new(each) }
+    if mtrrecs[0].type == 's'
+    	target = mtrrecs.shift.data
+	pingcount = 0
+	if target =~ /(\w+) (\d+)/
+		target = $1
+		pingcount = $2
+	end
+    end
+    id = SecureRandom.uuid
+    hops = Array.new
+    mtrrecs.each { |rec|
+	if rec.type == 'h'
+		hop = MtrHost.new(rec,pingcount,mtrrecs.select{|each| each.id == rec.id }).to_event_struct
+		if hops.size > 1
+			if (hops[hops.size - 1][:addr] != hop[:addr])
+				hops.push(hop)
+			else
+				# It's a duplicate of the last hop - drop it
+			end
+		else
+			hops.push(hop)
+		end
+	end
+    }
+    path = hops.collect {|each|each[:addr]}
+    pathsig = Digest::MD5.hexdigest(path.join('-'))
+    avgloss = hops.inject(0) {|loss,each| loss += each[:pingloss]} / path.size
+    avgrtt = hops.inject(0.0) {|rtt,each| rtt += each[:avgrtt]} / path.size
+    tracedata = { 	"id" => id,
+			"target" => target,
+			"message" => data ,
+			"hops" => hops,
+			"path" => path ,
+			"pathsig" => pathsig,
+			"pingcount"=>pingcount,
+			"avgloss"=>avgloss,
+			"avgrtt" => avgrtt,
+			"tags" => ["wholepath"]
+	}
+    yield LogStash::Event.new(tracedata)
+    # Construct a starting point for trace to target
+    yield LogStash::Event.new({
+	"id" => id,
+	"target" => target,
+	"tags" => ["hop"],
+	"seq" => -1,
+	"pathsig" => pathsig,
+	"A_node" => "TO:#{target}",
+	"Z_node" => hops[0][:addr],
+	"dns" => "startingpoint",
+	"avgrtt" => 0,
+	"avgloss" => 0
+    })
+    0.upto(path.size - 2) {
+       |index|
+       yield LogStash::Event.new({	"id" => id,
+					"target" => target,
+                                   	"tags" => ["hop"],
+					"pathsig" => pathsig,
+					"seq" => index,
+ 					"A_node" => hops[index][:addr],
+					"Z_node" => hops[index + 1][:addr],
+					"dns" => hops[index + 1][:dns],
+					"avgrtt" => hops[index + 1][:avgrtt],
+					"avgloss" => hops[index + 1][:avgloss]
+	})
+    }
+  end # def decode
+
+  # Encode a single event, this returns the raw data to be returned as a String
+  def encode_sync(event)
+    # Nothing to do.
+    @on_event.call(event, event)
+  end # def encode_sync
+
+end # class LogStash::Codecs::Mtrraw

data/logstash-codec-mtrraw.gemspec

@@ -0,0 +1,24 @@
+Gem::Specification.new do |s|
+  s.name          = 'logstash-codec-mtrraw'
+  s.version       = '0.1.0'
+  s.licenses      = ['Apache License (2.0)']
+  s.summary       = 'Converts optionally overloaded mtr --raw data to an event'
+  s.description   = 'Turn mtr --raw events with optional overloading into logstash events.  see docs'
+  s.homepage      = 'https://github.com/svdasein/logstash-codec-mtrraw'
+  s.authors       = ['svdasein']
+  s.email         = 'daveparker01@gmail.com'
+  s.require_paths = ['lib']
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
+   # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "codec" }
+
+  # Gem dependencies
+  s.add_runtime_dependency 'logstash-core-plugin-api', "~> 2.0"
+  s.add_runtime_dependency 'logstash-codec-line'
+  s.add_development_dependency 'logstash-devutils'
+end

data/spec/codecs/mtrraw_spec.rb

@@ -0,0 +1,3 @@
+# encoding: utf-8
+require_relative '../spec_helper'
+require "logstash/codecs/mtrraw"

data/spec/spec_helper.rb

@@ -0,0 +1,2 @@
+# encoding: utf-8
+require "logstash/devutils/rspec/spec_helper"

metadata

@@ -0,0 +1,100 @@
+--- !ruby/object:Gem::Specification
+name: logstash-codec-mtrraw
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- svdasein
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2017-11-27 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: logstash-core-plugin-api
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: logstash-codec-line
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: logstash-devutils
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Turn mtr --raw events with optional overloading into logstash events.  see
+  docs
+email: daveparker01@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CONTRIBUTORS
+- Gemfile
+- LICENSE
+- README.md
+- lib/logstash/codecs/mtrraw.rb
+- logstash-codec-mtrraw.gemspec
+- spec/codecs/mtrraw_spec.rb
+- spec/spec_helper.rb
+homepage: https://github.com/svdasein/logstash-codec-mtrraw
+licenses:
+- Apache License (2.0)
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: codec
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.5.1
+signing_key: 
+specification_version: 4
+summary: Converts optionally overloaded mtr --raw data to an event
+test_files:
+- spec/codecs/mtrraw_spec.rb
+- spec/spec_helper.rb
+has_rdoc: