logstash-input-beats 3.1.0.beta1-java

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 6dc83f50cea6b2105ecf0082caebe2040de2ab68
+  data.tar.gz: 19e6d17b3b0dd20c7664a44577d77b1bd032d2d4
+SHA512:
+  metadata.gz: bb9d9fb478c2a53368bf2b0cdcdc58b0955017d3a54ab42f5cb27e7afd516adba6e83f48f4d822e654d80ca75013f199e27ed5f8b51989b4728a980bbda910a6
+  data.tar.gz: 1723f186d35d767779c1ef14273e9822de1b49841896ca56bf47f105ab19cafc34d7d958d0e65c47b19c1c8d07d331415d172f41309cb2074e5b7ff86e30b8e6

data/CHANGELOG.md

@@ -0,0 +1,131 @@
+## 3.1.0
+    - Rewrite of the beats input in Java using the Netty framewwork, this rewrite is meant to be backward compatible with the previous implementation
+    but should yield better throughput and memory usage. https://github.com/logstash-plugins/logstash-input-beats/pull/93
+
+## 3.0.3
+
+  - Fix an issue when parsing multiple frames received from a filebeat client using pipelining.
+
+## 3.0.2
+
+  - relax constrains of `logstash-devutils` see https://github.com/elastic/logstash-devutils/issues/48
+
+## 3.0.1
+
+  - Republish all the gems under jruby.
+
+## 3.0.0
+
+  - Update the plugin to the version 2.0 of the plugin api, this change is required for Logstash 5.0 compatibility. See https://github.com/elastic/logstash/issues/5141
+
+## 2.2.8
+
+  - Fix #73 Bug in EventTransformCommon#codec_name, use config_name
+  - Add regression test for fix to #73
+  - Non deterministic error for the LSF integration test
+  - Make this plugin really a drop in replacement for the lumberjack input, so LSF can send their events to this plugin.
+
+## 2.2.7
+
+  - More robust test when using a random port #60
+  - Fix LSF integration tests #52
+
+## 2.2.6
+
+  - Do not use the identity map if we don't explicitly use the `multiline` codec
+
+## 2.2.5
+
+  - Fix failing tests introduce by the `ssl_key_passphrase` changes.
+  - Added an integration test for the `ssl_key_passphrase`
+  - Add an optional parameter for `auto_flush`
+
+## 2.2.4
+
+  - Fix bug where using `ssl_key_passphrase` wouldn't work 
+
+## 2.2.2
+
+  - Depend on logstash-core-plugin-api instead of logstash-core, removing the need to mass update plugins on major releases of logstash
+
+## 2.2.1
+
+  - New dependency requirements for logstash-core for the 5.0 release
+
+## 2.2.0
+
+  - The server can now do client side verification by providing a list of certificate authorities and configuring the `ssl_verify_mode`,
+    the server can use `peer`, if the client send a certificate it will be validated. Using `force_peer` will make sure the client provide a certificate
+    and it will be validated with the know CA.  #8
+
+## 2.1.4
+
+  - Change the `logger#warn` for `logger.debug` when a peer get disconnected, keep alive check from proxy can generate a lot of logs  #46
+
+## 2.1.3
+
+  - Make sure we stop all the threads after running the tests #48
+
+## 2.1.2
+
+  - Catch the `java.lang.InterruptedException` in the events broker
+  - Give a bit more time to the Thread to be started in the test #42
+
+## 2.1.1
+
+  - Release a new version of the gem that doesn't included any other gems, 2.1.0 is yanked from rubygems 
+
+## 2.1.0
+
+  - Refactor of the code to make it easier to unit test
+  - Fix a conncurrency error on high load on the SizeQueue #37
+  - Drop the internal SizeQueue to rely on Java Synchronous Queue
+  - Remove the majority of the nested blocks
+  - Move the CircuitBreaker inside an internal namespace so it doesn't conflict with the input lumberjack
+  - Add more debugging log statement
+  - Flush the codec when a disconnect happen
+  - Tag/Decorate the event when a shutdown occur.
+  - The name of the threads managed by the input beat are now meaningful.
+
+## 2.0.3
+
+  - Reduce the size of the gem by removing vendor jars
+
+## 2.0.2
+
+  - Copy the `beat.hostname` field into the `host` field for better compatibility with the other Logstash plugins #28
+  - Correctly merge multiple line with the multiline codec ref: #24
+
+## 2.0.0
+
+  - Add support for stream identity, the ID will be generated from beat.id+resource_id or beat.name + beat.source if not present #22 #13
+    The identity allow the multiline codec to correctly merge string from multiples files.
+
+## 0.9.6
+
+  - Fix an issue with rogue events created by buffered codecs #19
+
+## 0.9.5
+
+  - Set concurrent-ruby to 0.9.1 see https://github.com/elastic/logstash/issues/4141
+
+## 0.9.4
+
+  - Correctly decorate the event with the `add_field` and `tags` option from the config #12
+
+## 0.9.3
+
+  - Connection#run should rescue `Broken Pipe Error` #5
+  - Fix a `SystemCallErr` issue on windows when shutting down the server #9
+
+## 0.9.2
+
+  - fix an issue with the incorrectly calculated ack when the window_size was smaller than the ACK_RATIO see  https://github.com/logstash-plugins/logstash-input-beats/issues/3
+
+## 0.9.1
+
+  - Move the ruby-lumberjack library into the plugin
+
+## 0.9
+  - Created from `logstash-input-lumberjack` version 2.0.2 https://github.com/logstash-plugins/logstash-input-lumberjack
+  - Use SSL off by default

data/CONTRIBUTORS

@@ -0,0 +1,17 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Colin Surprenant (colinsurprenant)
+* Jordan Sissel (jordansissel)
+* Kurt Hurtado (kurtado)
+* Nick Ethier (nickethier)
+* Pier-Hugues Pellerin (ph)
+* Richard Pijnenburg (electrical)
+* Suyog Rao (suyograo)
+* Guy Boertje (guyboertje)
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in logstash-mass_effect.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,14 @@
+Copyright 2012–2016 Jordan Sissel, Elasticsearch and contributors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+

data/NOTICE.TXT

@@ -0,0 +1,5 @@
+Elasticsearch
+Copyright 2012-2015 Elasticsearch
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/PROTOCOL.md

@@ -0,0 +1,127 @@
+# The Lumberjack Protocol
+
+The needs that lead to this protocol are:
+
+* Encryption and Authentication to protect 
+* Compression should be used to reduce bandwidth
+* Round-trip latency should not damage throughput
+* Application-level message acknowledgement
+
+## Implementation Considerations
+
+# Lumberjack Protocol v1
+
+## Behavior
+
+Sequence and ack behavior (including sliding window, etc) is similar to TCP,
+but instead of bytes, messages are the base unit.
+
+A writer with a window size of 50 events can send up to 50 unacked events
+before blocking. A reader can acknowledge the 'last event' received to
+support bulk acknowledgements.
+
+Reliable, ordered byte transport is ensured by using TCP (or TLS on top), and
+this protocol aims to provide reliable, application-level, message transport.
+
+## Encryption and Authentication
+
+Currently this is to be handled by TLS.
+
+## Wire Format
+
+### Layering
+
+This entire protocol is built to be layered on top of TCP or TLS.
+
+### Framing
+
+      0                   1                   2                   3
+      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+     +---------------+---------------+-------------------------------+
+     |   version(1)  |   frame type  |     payload ...               |
+     +---------------------------------------------------------------+
+     |   payload continued...                                        |
+     +---------------------------------------------------------------+
+
+### 'data' frame type
+
+* SENT FROM WRITER ONLY
+* frame type value: ASCII 'D' aka byte value 0x44
+
+data is a map of string:string pairs. This is analogous to a Hash in Ruby, a
+JSON map, etc, but only strings are supported at this time.
+
+Payload:
+
+* 32bit unsigned sequence number
+* 32bit 'pair' count (how many key/value sequences follow)
+* 32bit unsigned key length followed by that many bytes for the key
+* 32bit unsigned value length followed by that many bytes for the value
+* repeat key/value 'count' times.
+
+Sequence number roll-over: If you receive a sequence number less than the
+previous value, this signals that the sequence number has rolled over.
+
+### 'json' frame type
+
+* SENT FROM WRITER ONLY
+* frame type value: ASCII 'J' aka byte value 0x4a
+
+data is json encoded.
+
+Payload:
+* 32bit unsigned sequence number
+* 32bit payload length (length in bytes of embedded json document)
+* 'length' bytes of json payload
+
+Sequence number roll-over: If you receive a sequence number less than the
+previous value, this signals that the sequence number has rolled over.
+
+### 'ack' frame type
+
+* SENT FROM READER ONLY
+* frame type value: ASCII 'A' aka byte value 0x41
+
+Payload:
+
+* 32bit unsigned sequence number.
+
+Bulk acks are supported. If you receive data frames in sequence order
+1,2,3,4,5,6, you can send an ack for '6' and the writer will take this to
+mean you are acknowledging all data frames before and including '6'.
+
+### 'window size' frame type
+
+* SENT FROM WRITER ONLY
+* frame type value: ASCII 'W' aka byte value 0x57
+
+Payload:
+
+* 32bit unsigned window size value in units of whole data frames.
+
+This frame is used to tell the reader the maximum number of unacknowledged
+data frames the writer will send before blocking for acks.
+
+### 'compressed' frame type
+
+* SENT FROM WRITER ONLY
+* frame type value: ASCII 'C' aka byte value 0x43
+
+Payload:
+
+* 32bit unsigned payload length 
+* 'length' bytes of compressed payload
+
+This frame type allows you to compress many frames into a single compressed
+envelope and is useful for efficiently compressing many small data frames.
+
+The compressed payload MUST contain full frames only, not partial frames.
+The uncompressed payload MUST be a valid frame stream by itself. As an example,
+you could have 3 data frames compressed into a single 'compressed' frame type:
+1D{k,v}{k,v}1D{k,v}{k,v}1D{k,v}{k,v} - when uncompressed, you should process
+the uncompressed payload as you would reading uncompressed frames from the
+network.
+
+TODO(sissel): It's likely this model is suboptimal, instead choose to
+use whole-stream compression z_stream in zlib (Zlib::ZStream in ruby) might be
+preferable.

data/README.md

@@ -0,0 +1,98 @@
+# Logstash Plugin
+
+[![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-input-beats.svg)](https://travis-ci.org/logstash-plugins/logstash-input-beats)
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+# Logstash 2.3 and higher
+bin/logstash-plugin install --no-verify
+
+# Prior to Logstash 2.3
+bin/plugin install --no-verify
+
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+# Logstash 2.3 and higher
+bin/logstash-plugin install --no-verify
+
+# Prior to Logstash 2.3
+bin/plugin install --no-verify
+
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/VERSION

@@ -0,0 +1 @@
+3.1.0.beta1

data/lib/logstash-input-beats_jars.rb

@@ -0,0 +1,17 @@
+# AUTOGENERATED BY THE GRADLE SCRIPT. DO NOT EDIT.
+
+require 'jar_dependencies'
+require_jar('org.apache.logging.log4j', 'log4j-1.2-api', '2.6.1')
+require_jar('org.apache.logging.log4j', 'log4j-slf4j-impl', '2.6.1')
+require_jar('org.bouncycastle', 'bcprov-jdk15on', '1.54')
+require_jar('org.bouncycastle', 'bcpkix-jdk15on', '1.54')
+require_jar('io.netty', 'netty-all', '4.1.1.Final')
+require_jar('io.netty', 'netty-tcnative-boringssl-static', '1.1.33.Fork17')
+require_jar('org.apache.logging.log4j', 'log4j-api', '2.6.1')
+require_jar('org.apache.logging.log4j', 'log4j-core', '2.6.1')
+require_jar('org.javassist', 'javassist', '3.20.0-GA')
+require_jar('com.fasterxml.jackson.core', 'jackson-core', '2.7.5')
+require_jar('com.fasterxml.jackson.core', 'jackson-annotations', '2.7.5')
+require_jar('com.fasterxml.jackson.core', 'jackson-databind', '2.7.5')
+require_jar('com.fasterxml.jackson.module', 'jackson-module-afterburner', '2.7.5')
+require_jar('org.logstash.beats', 'logstash-input-beats', '3.1.0.beta1')

data/lib/logstash/inputs/beats.rb

@@ -0,0 +1,184 @@
+# encoding: utf-8
+require "logstash/inputs/base"
+require "logstash/namespace"
+require "logstash/timestamp"
+require "logstash/codecs/identity_map_codec"
+require "logstash/codecs/multiline"
+require "logstash/util"
+require "logstash-input-beats_jars"
+
+import "org.logstash.beats.Server"
+import "org.logstash.netty.SslSimpleBuilder"
+import "org.logstash.netty.PrivateKeyConverter"
+import "java.io.FileInputStream"
+
+# This input plugin enables Logstash to receive events from the
+# https://www.elastic.co/products/beats[Elastic Beats] framework.
+#
+class LogStash::Codecs::Base
+  # This monkey patch add callback based
+  # flow to the codec until its shipped with core.
+  # This give greater flexibility to the implementation by
+  # sending more data to the actual block.
+  if !method_defined?(:accept)
+    def accept(listener)
+      decode(listener.data) do |event|
+        listener.process_event(event)
+      end
+    end
+  end
+  if !method_defined?(:auto_flush)
+    def auto_flush(*)
+    end
+  end
+end
+
+class LogStash::Inputs::Beats < LogStash::Inputs::Base
+  require "logstash/inputs/beats/codec_callback_listener"
+  require "logstash/inputs/beats/event_transform_common"
+  require "logstash/inputs/beats/decoded_event_transform"
+  require "logstash/inputs/beats/raw_event_transform"
+  require "logstash/inputs/beats/message_listener"
+  require "logstash/inputs/beats/tls"
+
+  config_name "beats"
+
+  default :codec, "plain"
+
+  # The IP address to listen on.
+  config :host, :validate => :string, :default => "0.0.0.0"
+
+  # The port to listen on.
+  config :port, :validate => :number, :required => true
+
+  # Events are by default sent in plain text. You can
+  # enable encryption by setting `ssl` to true and configuring
+  # the `ssl_certificate` and `ssl_key` options.
+  config :ssl, :validate => :boolean, :default => false
+
+  # SSL certificate to use.
+  config :ssl_certificate, :validate => :path
+
+  # SSL key to use.
+  config :ssl_key, :validate => :path
+
+  # SSL key passphrase to use.
+  config :ssl_key_passphrase, :validate => :password
+
+  # Validate client certificates against these authorities. 
+  # You can define multiple files or paths. All the certificates will
+  # be read and added to the trust store. You need to configure the `ssl_verify_mode`
+  # to `peer` or `force_peer` to enable the verification.
+  #
+  # This feature only supports certificates that are directly signed by your root CA.
+  # Intermediate CAs are currently not supported.
+  # 
+  config :ssl_certificate_authorities, :validate => :array, :default => []
+
+  # By default the server doesn't do any client verification.
+  # 
+  # `peer` will make the server ask the client to provide a certificate. 
+  # If the client provides a certificate, it will be validated.
+  #
+  # `force_peer` will make the server ask the client to provide a certificate.
+  # If the client doesn't provide a certificate, the connection will be closed.
+  #
+  # This option needs to be used with `ssl_certificate_authorities` and a defined list of CAs.
+  config :ssl_verify_mode, :validate => ["none", "peer", "force_peer"], :default => "none"
+
+  # The number of seconds before we raise a timeout. 
+  # This option is useful to control how much time to wait if something is blocking the pipeline.
+  config :congestion_threshold, :validate => :number, :default => 5
+
+  # This is the default field to which the specified codec will be applied.
+  config :target_field_for_codec, :validate => :string, :default => "message", :deprecated => "This option is now deprecated, the plugin is now compatible with Filebeat and Logstash-Forwarder"
+
+  # The minimum TLS version allowed for the encrypted connections. The value must be one of the following:
+  # 1.0 for TLS 1.0, 1.1 for TLS 1.1, 1.2 for TLS 1.2
+  config :tls_min_version, :validate => :number, :default => TLS.min.version
+
+  # The maximum TLS version allowed for the encrypted connections. The value must be the one of the following:
+  # 1.0 for TLS 1.0, 1.1 for TLS 1.1, 1.2 for TLS 1.2
+  config :tls_max_version, :validate => :number, :default => TLS.max.version
+
+  # The list of ciphers suite to use, listed by priorities.
+  config :cipher_suites, :validate => :array, :default => org.logstash.netty.SslSimpleBuilder::DEFAULT_CIPHERS
+
+  def register
+    if !@ssl
+      @logger.warn("Beats input: SSL Certificate will not be used") unless @ssl_certificate.nil?
+      @logger.warn("Beats input: SSL Key will not be used") unless @ssl_key.nil?
+    elsif !ssl_configured?
+      raise LogStash::ConfigurationError, "Certificate or Certificate Key not configured"
+    end
+
+    @logger.info("Beats inputs: Starting input listener", :address => "#{@host}:#{@port}")
+
+    # wrap the configured codec to support identity stream
+    # from the producers if running with the multiline codec.
+    #
+    # If they dont need an identity map, codec are stateless and can be reused
+    # accross multiples connections.
+    if need_identity_map?
+      @codec = LogStash::Codecs::IdentityMapCodec.new(@codec)
+    end
+
+    @server = create_server
+  end # def register
+
+  def create_server
+    server = org.logstash.beats.Server.new(@port)
+    if @ssl
+      private_key_converter = org.logstash.netty.PrivateKeyConverter.new(ssl_key, ssl_key_passphrase)
+      ssl_builder = org.logstash.netty.SslSimpleBuilder.new(FileInputStream.new(ssl_certificate), private_key_converter.convert(), ssl_key_passphrase)
+        .setProtocols(convert_protocols)
+        .setCipherSuites(normalized_ciphers)
+
+      if client_authentification?
+        if @ssl_verify_mode.upcase == "FORCE_PEER"
+            ssl_builder.setVerifyMode(org.logstash.netty.SslSimpleBuilder::SslClientVerifyMode::FORCE_PEER)
+        end
+
+        ssl_builder.setCertificateAuthorities(@ssl_certificate_authorities)
+      end
+
+      server.enableSSL(ssl_builder)
+    end
+
+    server
+  end
+
+  def ssl_configured?
+    !(@ssl_certificate.nil? || @ssl_key.nil?)
+  end
+
+  def target_codec_on_field?
+    !@target_codec_on_field.empty?
+  end
+
+  def run(output_queue)
+    message_listener = MessageListener.new(output_queue, self)
+    @server.setMessageListener(message_listener)
+    @server.listen
+  end # def run
+
+  def stop
+    @server.stop
+  end
+
+  def need_identity_map?
+    @codec.kind_of?(LogStash::Codecs::Multiline)
+  end
+
+  def client_authentification?
+    @ssl_certificate_authorities && @ssl_certificate_authorities.size > 0
+  end
+
+  def normalized_ciphers
+    @cipher_suites.map(&:upcase)
+  end
+
+  def convert_protocols
+    TLS.get_supported(@tls_min_version..@tls_max_version).map(&:name)
+  end
+end