logstash-input-snmp 0.1.0.beta1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: d344864bfc29473393a8d06527a52b5a606efcb8384ce300751898be6daec7ca
+  data.tar.gz: 3fdf527c93de3d59a6f7c3f20e07e3bfc02d893706b649d2ee7c5e310097e7d0
+SHA512:
+  metadata.gz: 8d2dc6418f15151c5ade6a900fb28f352020c78e9b95b579b9d8a594f84f539e6bb65570c074f41d9b7ed4c5ba96c6ffac5a745512094af74cb2f3860e0cc277
+  data.tar.gz: 827ed778277f612c1e99982bd8a039d99fcc87b9221f7b26e400c1a672d1b2cf12c21e4a83f4a32ca208a619db9ae31a8fac468d249a3b8e2ee47c6f76bc5a76

data/CHANGELOG.md

@@ -0,0 +1,3 @@
+## 0.1.0.beta1
+  - First beta version
+

data/CONTRIBUTORS

@@ -0,0 +1,10 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Colin Surprenant - colin.surprenant@gmail.com
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/DEVELOPER.md

@@ -0,0 +1,2 @@
+# logstash-input-snmp
+Example input plugin. This should help bootstrap your effort to write your own input plugin!

data/Gemfile

@@ -0,0 +1,11 @@
+source 'https://rubygems.org'
+
+gemspec
+
+logstash_path = ENV["LOGSTASH_PATH"] || "../../logstash"
+use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "1"
+
+if Dir.exist?(logstash_path) && use_logstash_source
+  gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
+  gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
+end

data/LICENSE

@@ -0,0 +1,11 @@
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,86 @@
+# Logstash Plugin
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+bin/logstash-plugin install --no-verify
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/logstash-plugin install /your/local/plugin/logstash-filter-awesome.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash-input-snmp_jars.rb

@@ -0,0 +1,4 @@
+# AUTOGENERATED BY THE GRADLE SCRIPT. DO NOT EDIT.
+
+require 'jar_dependencies'
+require_jar('org.snmp4j', 'snmp4j', '2.5.11')

data/lib/logstash/inputs/snmp.rb

@@ -0,0 +1,158 @@
+# encoding: utf-8
+require "logstash/inputs/base"
+require "logstash/namespace"
+require "stud/interval"
+require "socket" # for Socket.gethostname
+require_relative "snmp/client"
+require_relative "snmp/mib"
+
+# Generate a repeating message.
+#
+# This plugin is intented only as an example.
+
+class LogStash::Inputs::Snmp < LogStash::Inputs::Base
+  config_name "snmp"
+
+  # List of OIDs for which we want to retrieve the scalar value
+  config :get,:validate => :array # ["1.3.6.1.2.1.1.1.0"]
+
+  # List of OIDs for which we want to retrieve the subtree of information
+  config :walk,:validate => :array # ["1.3.6.1.2.1.1.1.0"]
+
+  # List of hosts to query the configured `get` and `walk` options.
+  #
+  # Each host definition is a hash and must define the `host` key and value.
+  #  `host` must use the format {tcp|udp}:{ip address}/{port}
+  #  for example `host => "udp:127.0.0.1/161"`
+  # Each host definition can optionally include the following keys and values:
+  #  `community` with a default value of `public`
+  #  `version` with a default value of `2c`
+  #  `retries` with a detault value of `2`
+  #  `timeout` in milliseconds with a default value of `1000`
+  config :hosts, :validate => :array  #[ {"host" => "udp:127.0.0.1/161", "community" => "public"} ]
+
+  # List of paths of MIB .dic files of dirs. If a dir path is specified, all files with .dic extension will be loaded.
+  #
+  # ATTENTION: a MIB .dic file must be generated using the libsmi library `smidump` command line utility
+  # like this for example. Here the `RFC1213-MIB.txt` file is an ASN.1 MIB file.
+  #
+  # `$ smidump -k -f python RFC1213-MIB.txt > RFC1213-MIB.dic`
+  #
+  # The OSS libsmi library https://www.ibr.cs.tu-bs.de/projects/libsmi/ is available & installable
+  # on most OS.
+  config :mib_paths, :validate => :array # ["path/to/mib.dic", "path/to/mib/dir"]
+
+  # number of OID root digits to ignore in event field name. For example, in a numeric OID
+  # like 1.3.6.1.2.1.1.1.0" the first 5 digits could be ignored by setting oid_root_skip => 5
+  # which would result in a field name "1.1.1.0". Similarly when a MIB is used an OID such
+  # as "1.3.6.1.2.mib-2.system.sysDescr.0" would become "mib-2.system.sysDescr.0"
+  config :oid_root_skip, :validate => :number, :default => 0
+
+  # Set polling interval in seconds
+  #
+  # The default, `30`, means poll each host every 30second.
+  config :interval, :validate => :number, :default => 30
+
+  def register
+    validate_oids!
+    validate_hosts!
+
+    mib = LogStash::SnmpMib.new
+    Array(@mib_paths).each do |path|
+      # TODO handle errors
+      mib.add_mib_path(path)
+    end
+
+    @client_definitions = []
+    @hosts.each do |host|
+      host_name = host["host"]
+      community = host["community"] || "public"
+      version = host["version"] || "2c"
+      retries = host["retries"] || 2
+      timeout = host["timeout"] || 1000
+
+      definition = {
+        :client => LogStash::SnmpClient.new(host_name, community, version, retries, timeout, mib),
+        :get => Array(get),
+        :walk => Array(walk),
+      }
+      @client_definitions << definition
+    end
+  end
+
+  def run(queue)
+    # for now a naive single threaded poller which sleeps for the given interval between
+    # each run. each run polls all the defined hosts for the get and walk options.
+    while !stop?
+      @client_definitions.each do |definition|
+        result = {}
+        if !definition[:get].empty?
+          begin
+            result = result.merge(definition[:client].get(definition[:get], @oid_root_skip))
+          rescue => e
+            logger.error("error invoking get operation on OIDs: #{definition[:get]}, ignoring", :exception => e, :backtrace => e.backtrace)
+          end
+        end
+        if  !definition[:walk].empty?
+          definition[:walk].each do |oid|
+            begin
+              result = result.merge(definition[:client].walk(oid, @oid_root_skip))
+            rescue => e
+              logger.error("error invoking walk operation on OID: #{oid}, ignoring", :exception => e, :backtrace => e.backtrace)
+            end
+          end
+        end
+
+        unless result.empty?
+          event = LogStash::Event.new(result)
+          decorate(event)
+          queue << event
+        end
+      end
+
+      Stud.stoppable_sleep(@interval) { stop? }
+    end
+  end
+
+  def stop
+  end
+
+  private
+
+  OID_REGEX = /^\.?([0-9\.]+)$/
+  HOST_REGEX = /^(udp|tcp):.+\/\d+$/i
+
+  def validate_oids!
+    @get = Array(@get).map do |oid|
+      # verify oids for valid pattern and get rid or any leading dot if present
+      unless oid =~ OID_REGEX
+        raise(LogStash::ConfigurationError, "The get option oid '#{oid}' has an invalid format")
+      end
+      $1
+    end
+
+    @walk = Array(@walk).map do |oid|
+      # verify oids for valid pattern and get rid or any leading dot if present
+      unless oid =~ OID_REGEX
+        raise(LogStash::ConfigurationError, "The walk option oid '#{oid}' has an invalid format")
+      end
+      $1
+    end
+
+    raise(LogStash::ConfigurationError, "at least one get OID or one walk OID is required") if @get.empty? && @walk.empty?
+  end
+
+  def validate_hosts!
+    # TODO: for new we only validate the host part, not the other optional options
+
+    raise(LogStash::ConfigurationError, "at least one host definition is required") if Array(@hosts).empty?
+
+    @hosts.each do |host|
+      raise(LogStash::ConfigurationError, "each host definition must have a \"host\" option") if !host.is_a?(Hash) || host["host"].nil?
+      unless host["host"] =~ HOST_REGEX
+        raise(LogStash::ConfigurationError, "invalid host option format")
+      end
+      raise(LogStash::ConfigurationError, "tcp is not yet supported, only udp") if $1 =~ /tcp/i
+    end
+  end
+end

data/lib/logstash/inputs/snmp/client.rb

@@ -0,0 +1,135 @@
+require "java"
+require "logstash-input-snmp_jars.rb"
+
+java_import "org.snmp4j.CommunityTarget"
+java_import "org.snmp4j.PDU"
+java_import "org.snmp4j.Snmp"
+java_import "org.snmp4j.Target"
+java_import "org.snmp4j.TransportMapping"
+java_import "org.snmp4j.event.ResponseEvent"
+java_import "org.snmp4j.mp.SnmpConstants"
+java_import "org.snmp4j.smi.Address"
+java_import "org.snmp4j.smi.GenericAddress"
+java_import "org.snmp4j.smi.OID"
+java_import "org.snmp4j.smi.OctetString"
+java_import "org.snmp4j.smi.VariableBinding"
+java_import "org.snmp4j.transport.DefaultUdpTransportMapping"
+java_import "org.snmp4j.util.TreeUtils"
+java_import "org.snmp4j.util.DefaultPDUFactory"
+java_import "org.snmp4j.asn1.BER"
+
+module LogStash
+  class SnmpClientError < StandardError
+  end
+
+  class SnmpClient
+
+    def initialize(address, community, version, retries, timeout, mib)
+      @target = build_target(address, community, version, retries, timeout)
+      @mib = mib
+
+      # for now hardwired udp transport
+      transport = DefaultUdpTransportMapping.new
+      @snmp = Snmp.new(transport)
+      transport.listen()
+    end
+
+    def get(oids, strip_root = 0)
+      pdu = PDU.new
+      Array(oids).each { |oid| pdu.add(VariableBinding.new(OID.new(oid))) }
+      pdu.setType(PDU::GET)
+
+      response_event = @snmp.send(pdu, @target, nil)
+      return nil if response_event.nil?
+
+      e = response_event.getError
+      raise(SnmpClientError, "error sending snmp get request: #{e.inspect}, #{e.getMessage}") if e
+
+      result = {}
+      response_pdu = response_event.getResponse
+      raise(SnmpClientError, "timeout sending snmp get request") if response_pdu.nil?
+
+      size = response_pdu.size
+      (0..size - 1).each do |i|
+        variable_binding = response_pdu.get(i)
+        oid = variable_binding.getOid.toString
+        variable = variable_binding.getVariable
+        value = coerce(variable)
+
+        result[@mib.map_oid(oid, strip_root)] = value
+      end
+
+      result
+    end
+
+
+    def walk(oid, strip_root = 0)
+      result = {}
+      treeUtils = TreeUtils.new(@snmp, DefaultPDUFactory.new)
+      events = treeUtils.getSubtree(@target, OID.new(oid))
+      return nil if events.nil? || events.size == 0
+
+      events.each do |event|
+        next if event.nil?
+
+        if event.isError
+          # TODO: see if we can salvage non errored event here
+          raise(SnmpClientError, "error sending snmp walk request: #{event.getErrorMessage}")
+        end
+
+        var_bindings = event.getVariableBindings
+        next if var_bindings.nil? || var_bindings.size == 0
+
+        var_bindings.each do |var_binding|
+          next if var_binding.nil?
+
+          oid = var_binding.getOid.toString
+          variable = var_binding.getVariable
+          value = coerce(variable)
+
+          result[@mib.map_oid(oid, strip_root)] = value
+         end
+      end
+
+      result
+    end
+
+    private
+
+    def coerce(variable)
+      variable_syntax = variable.getSyntax
+      # puts("variable.getSyntaxString=#{variable.getSyntaxString}")
+      case variable_syntax
+      when BER::OCTETSTRING
+        variable.toString
+      when BER::TIMETICKS, BER::COUNTER, BER::COUNTER32
+        variable.toLong
+      when BER::INTEGER, BER::INTEGER32, BER::GAUGE, BER::GAUGE32
+        variable.toInt
+      when BER::IPADDRESS
+        variable.toString
+      when BER::OID
+        variable.toString
+      when BER::NOSUCHOBJECT
+        "Error: No Such Instance currently exists at this OID"
+      else
+        raise(SnmpClientError, "unknown variable syntax #{variable_syntax}, #{variable.getSyntaxString}")
+      end
+    end
+
+    def build_target(address, community, version, retries, timeout)
+      target = CommunityTarget.new
+      target.setCommunity(OctetString.new(community))
+      target.setAddress(GenericAddress.parse(address))
+      target.setRetries(retries)
+      target.setTimeout(timeout)
+      target.setVersion(parse_version(version))
+      target
+    end
+
+    def parse_version(version)
+      # TODO implement
+      SnmpConstants.version2c
+    end
+  end
+end

data/lib/logstash/inputs/snmp/mib.rb

@@ -0,0 +1,159 @@
+# encoding: utf-8
+
+module LogStash
+  class SnmpMibError < StandardError
+  end
+
+  class SnmpMib
+    attr_reader :tree
+
+    class Oid
+      def self.parse(oid)
+        oid.split(".").map(&:to_i)
+      end
+    end
+
+    class BaseNode
+      attr_reader :name, :childs
+
+      def initialize(name)
+        @name = name
+        @childs = []
+      end
+    end
+
+    class Node < BaseNode
+      attr_reader :node_type, :module_name, :oid, :oid_path
+
+      def initialize(node_type, name, module_name, oid)
+        super(name)
+        @node_type = node_type
+        @module_name = module_name
+        @oid = oid
+        @oid_path = Oid.parse(oid)
+      end
+    end
+
+    class Tree
+      def initialize
+        @root = BaseNode.new("root")
+      end
+
+      def add_node(node)
+        warnings = []
+        current = @root
+        path = node.oid_path.dup
+
+        # follow the OID path up until but not including the last node
+        # and add intermediate missing nodes if needed
+        last_node = path.pop
+        path.each do |i|
+          if current.childs[i].nil?
+            current.childs[i] = BaseNode.new(i.to_s)
+          end
+          current = current.childs[i]
+        end
+
+        if current.childs[last_node] && current.childs[last_node].name != node.name
+          warnings << "warning: overwriting MIB OID '#{node.oid}' and name '#{current.childs[last_node].name}' with new name '#{node.name}' from module '#{node.module_name}'"
+        end
+        current.childs[last_node] = node
+
+        warnings
+      end
+
+      def map_oid(oid, strip_root = 0)
+        path = Oid.parse(oid)
+
+        result = []
+        node = @root
+
+        loop do
+          break if path.empty?
+          i = path.shift
+
+          node = node.childs[i]
+
+          if node.nil?
+            result += path.unshift(i)
+            break
+          end
+          result << node.name
+        end
+
+        result.drop(strip_root).join(".")
+      end
+    end
+
+    def initialize
+      @tree = Tree.new
+    end
+
+    # add a specific mib dic file or all mib dic files of the given directory to the current mib database
+    # @param path [String] a file or directory path to mib dic file(s)
+    # @return [Array] array of warning strings if any OID or name has been overwritten or the empty array when no warning
+    def add_mib_path(path)
+      dic_files = if ::File.directory?(path)
+        Dir[::File.join(path, "*.dic")]
+      elsif ::File.file?(path)
+        [path]
+      else
+        raise(SnmpMibError, "file or directory path expected: #{path.to_s}")
+      end
+
+      warnings = []
+      dic_files.each do |f|
+        module_name, nodes = read_mib_dic(f)
+
+        nodes.each do |k, v|
+          warnings += @tree.add_node(Node.new(v["nodetype"], k, v["moduleName"], v["oid"]))
+        end
+      end
+
+      warnings
+    end
+
+    # read and parse a mib dic file
+    #
+    # @param filename [String] file path of a mib dic file
+    # @return [[String, Hash, Hash, Hash]] the 2-tuple of the mib module name and the complete nodes
+    def read_mib_dic(filename)
+      mib = eval_mib_dic(filename)
+      raise(SnmpMibError, "invalid mib dic format for file #{filename}") unless mib
+      module_name = mib["moduleName"]
+      raise(SnmpMibError, "invalid mib dic format for file #{filename}") unless module_name
+      nodes = mib["nodes"]
+      raise(SnmpMibError, "no nodes defined in mib dic file #{filename}") unless nodes
+
+      # name_hash is { mib-name => oid }
+      # name_hash = {}
+      # nodes.each { |k, v| name_hash[k] = v["oid"] }
+      # if mib["notifications"]
+      #   mib["notifications"].each { |k, v| name_hash[k] = v["oid"] }
+      # end
+
+      [module_name, nodes]
+    end
+
+    def map_oid(oid, strip_root = 0)
+      @tree.map_oid(oid, strip_root)
+    end
+
+    private
+
+    def eval_mib_dic(filename)
+      mib_dic = IO.read(filename)
+      mib_hash = mib_dic.
+        gsub(':', '=>').                  # fix hash syntax
+        gsub('(', '[').gsub(')', ']').    # fix tuple syntax
+        sub('FILENAME =', 'filename =').  # get rid of constants
+        sub('MIB =', 'mib =')
+
+      mib = nil
+      eval(mib_hash)
+      mib
+    rescue => e
+      raise(SnmpMibError, "error parsing mib dic file: #{filename}, error: #{e.message}")
+    end
+  end
+end