logstash-codec-joinlines 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: cff8b92826a48b224f39c357774bf38c576d084fc4c367dfeab8235fb2b8df69
+  data.tar.gz: c3788765ed29c61f7bc3916ffdea4289e8401f9f54454b8478be467f4126482a
+SHA512:
+  metadata.gz: 8e95490d1e38db256450e56ce14b16e3d7dfa380b8178cb2fd71d2c0af52d9039a459864d89a6980096caa3007cdb73a669f3991d87143cad6c2d6b8f527bc24
+  data.tar.gz: 32b1604e0071833000c9414b1654b6852a333189894c4b0a22fad9a5a5c90cdf40733268af14d4a2a22538bd96c11d23a1d903133843f4d6c38dfcfbc1217611

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+## 0.1.0
+  - Plugin created with the logstash plugin generator
+  - Copied code from logstash-plugin-multiline
+  - Implemented multiple patterns
+  

data/CONTRIBUTORS

@@ -0,0 +1,19 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Developers of logstash-codec-multiline
+    * Colin Surprenant (colinsurprenant)
+    * Jordan Sissel (jordansissel)
+    * João Duarte (jsvd)
+    * Kurt Hurtado (kurtado)
+    * Pier-Hugues Pellerin (ph)
+    * Richard Pijnenburg (electrical)
+    * Suyog Rao (suyograo)
+    * Guy Boertje (guyboertje)
+* Svein L. Ellingsen (lovmoen)
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/DEVELOPER.md

@@ -0,0 +1,3 @@
+# logstash-codec-joinlines
+A codec plugin for Logstash; based on logstash-codec-multiline.
+It allows for specifying multiple patterns to match for joining lines.

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,11 @@
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,86 @@
+# Logstash Plugin
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-codec-awesome", :path => "/your/local/logstash-codec-awesome"
+```
+- Install plugin
+```sh
+bin/logstash-plugin install --no-verify
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'codec {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-codec-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/logstash-plugin install /your/local/plugin/logstash-codec-awesome.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/codecs/auto_flush.rb

@@ -0,0 +1,48 @@
+# encoding: utf-8
+require "concurrent"
+require "logstash/codecs/retriggerable_task"
+
+module LogStash module Codecs class AutoFlush
+  def initialize(mc, interval)
+    @mc, @interval = mc, interval
+    @stopped = Concurrent::AtomicBoolean.new # false by default
+    @task = RetriggerableTask.new(@interval, self)
+  end
+
+  def timeout
+    @mc.auto_flush
+  end
+
+  def start
+    # can't start if pipeline is stopping
+    return self if stopped?
+    @task.retrigger
+    self
+  end
+
+  def stopped?
+    @stopped.value
+  end
+
+  def stop
+    @stopped.make_true
+    @task.close
+  end
+end
+
+class AutoFlushUnset
+  def initialize(mc, interval)
+  end
+
+  def stopped?
+    true
+  end
+
+  def start
+    self
+  end
+
+  def stop
+    self
+  end
+end end end

data/lib/logstash/codecs/identity_map_codec.rb

@@ -0,0 +1,347 @@
+# encoding: utf-8
+require "logstash/namespace"
+require "thread_safe"
+require "concurrent"
+
+# This class is a Codec duck type
+# Using Composition, it maps from a stream identity to
+# a cloned codec instance via the same API as a Codec
+# it implements the codec public API
+
+module LogStash module Codecs class IdentityMapCodec
+  # subclass of Exception, LS has more than limit (20000) active streams
+  class IdentityMapUpperLimitException < Exception; end
+
+  module EightyPercentWarning
+    extend self
+    def visit(imc)
+      current_size, limit = imc.current_size_and_limit
+      return if current_size < (limit * 0.8)
+      imc.logger.warn("IdentityMapCodec has reached 80% capacity",
+        :current_size => current_size, :upper_limit => limit)
+    end
+  end
+
+  module UpperLimitReached
+    extend self
+    def visit(imc)
+      current_size, limit = imc.current_size_and_limit
+      return if current_size < limit
+      # we hit the limit
+      # try to clean out stale streams
+      current_size, limit = imc.map_cleanup
+      return if current_size < limit
+      # we are still at the limit and all streams are in use
+      imc.logger.error("IdentityMapCodec has reached 100% capacity",
+          :current_size => current_size, :upper_limit => limit)
+      raise IdentityMapUpperLimitException.new
+    end
+  end
+
+  class PeriodicRunner
+    def initialize(listener, interval, method_symbol)
+      @listener, @interval = listener, interval
+      @method_symbol = method_symbol
+      @running = Concurrent::AtomicBoolean.new(false)
+    end
+
+    def start
+      return self if running?
+      @running.make_true
+      @thread = Thread.new() do
+        while running? do
+          sleep @interval
+          break if !running?
+          @listener.send(@method_symbol)
+        end
+      end
+      self
+    end
+
+    def running?
+      @running.value
+    end
+
+    def stop
+      return if !running?
+      @running.make_false
+      if @thread.alive?
+        @thread.wakeup
+        @thread.join
+      end
+      @listener = nil
+    end
+  end
+
+  class NoopRunner
+    attr_reader :start, :stop
+    def running?() false; end
+  end
+
+  # A composite class to hold both the codec, the eviction_timeout and a last_used timestamp
+  # instances of this Value Object are stored in the mapping hash
+  class CodecValue
+    attr_reader :codec
+    attr_accessor :eviction_timeout, :auto_flush_timeout
+
+    def initialize(codec)
+      @codec = codec
+    end
+  end
+
+  #maximum size of the mapping hash
+  MAX_IDENTITIES = 20_000
+
+  # time after which a stream is
+  # considered stale
+  # each time a stream is accessed
+  # it is given a new timeout
+  EVICT_TIMEOUT = 60 * 60 * 1 # 1 hour
+
+  # time that the cleaner thread sleeps for
+  # before it tries to clean out stale mappings
+  CLEANER_INTERVAL = 60 * 5 # 5 minutes
+
+  attr_reader :identity_map
+  attr_accessor :base_codec, :cleaner, :auto_flusher
+
+  def initialize(codec)
+    @base_codec = codec
+    @base_codecs = [codec]
+    @identity_map = ThreadSafe::Hash.new &method(:codec_builder)
+    @max_identities = MAX_IDENTITIES
+    @evict_timeout = EVICT_TIMEOUT
+    cleaner_interval(CLEANER_INTERVAL)
+    if codec.respond_to?(:use_mapper_auto_flush) &&
+        (@auto_flush_interval = codec.use_mapper_auto_flush)
+      @auto_flusher = PeriodicRunner.new(self, 0.5, :auto_flush_mapped)
+    else
+      @auto_flusher = NoopRunner.new
+    end
+
+    @decode_block = lambda {|*| true }
+    @eviction_block = nil
+  end
+
+  # ==============================================
+  # Constructional/builder methods
+  # chain this method off of new
+  #
+  # used to add a non-default maximum identities
+  def max_identities(max)
+    @max_identities = max.to_i
+    self
+  end
+
+  # used to add a non-default evict timeout
+  def evict_timeout(timeout)
+    @evict_timeout = timeout.to_i
+    self
+  end
+
+  # used to add  a non-default cleaner interval
+  def cleaner_interval(interval)
+    @cleaner.stop if @cleaner
+    @cleaner = PeriodicRunner.new(self, interval.to_i, :map_cleanup)
+    self
+  end
+
+  # used to add  a non-default eviction block
+  def eviction_block(block)
+    @eviction_block = block
+    self
+  end
+
+  # end Constructional/builder methods
+  # ==============================================
+
+  # ==============================================
+  # IdentityMapCodec API
+  def evict(identity)
+    # maybe called more than once
+    if (compo = identity_map.delete(identity))
+      compo.codec.auto_flush if compo.codec.respond_to?(:auto_flush)
+    end
+  end
+  # end IdentityMapCodec API
+  # ==============================================
+
+  # ==============================================
+  # Codec API
+  def decode(data, identity = nil, &block)
+    @decode_block = block if @decode_block != block
+    stream_codec(identity).decode(data, &block)
+  end
+
+  def accept(listener)
+    stream_codec(listener.path).accept(listener)
+  end
+
+  alias_method :<<, :decode
+
+  def encode(event, identity = nil)
+    stream_codec(identity).encode(event)
+  end
+
+  def flush(&block)
+    all_codecs.each do |codec|
+      #let ruby do its default args thing
+      if block_given?
+        codec.flush(&block)
+      else
+        if codec.respond_to?(:auto_flush)
+          codec.auto_flush
+        else
+          #try this, no guarantees
+          codec.flush
+        end
+      end
+    end
+  end
+
+  def close()
+    cleaner.stop
+    auto_flusher.stop
+    all_codecs.each(&:close)
+  end
+  # end Codec API
+  # ==============================================
+
+  def auto_flush_mapped
+    if !identity_count.zero?
+      nowf = Time.now.to_f
+      identity_map.each do |identity, compo|
+        next if compo.auto_flush_timeout.zero?
+        next unless nowf > compo.auto_flush_timeout
+        compo.codec.auto_flush
+        # at eof (tail and read) no more lines for a while or ever
+        # so reset compo.auto_flush_timeout
+        compo.auto_flush_timeout = 0
+      end
+    end
+  end
+
+  def flush_mapped(listener)
+    listener_has_path = listener.respond_to?(:path)
+    identity_map.each do |identity, compo|
+      listener.path = identity if listener_has_path
+      compo.codec.auto_flush(listener)
+    end
+  end
+
+  def all_codecs
+    no_streams? ? @base_codecs : identity_map.values.map(&:codec)
+  end
+
+  def max_limit
+    @max_identities
+  end
+
+  def identity_count
+    identity_map.size
+  end
+
+  # support cleaning of stale stream/codecs
+  # a stream is considered stale if it has not
+  # been accessed in the last @evict_timeout
+  # period (default 1 hour)
+  def map_cleanup
+    if !identity_count.zero?
+      nowi = Time.now.to_i
+      # delete_if is atomic
+      # contents should not mutate during this call
+      identity_map.delete_if do |identity, compo|
+        if (flag = compo.eviction_timeout <= nowi)
+          evict_flush(compo.codec)
+        end
+        flag
+      end
+    end
+    current_size_and_limit
+  end
+
+  def evict_flush(codec)
+    if codec.respond_to?(:auto_flush)
+      codec.auto_flush
+    else
+      if (block = @eviction_block || @decode_block)
+        codec.flush(&block)
+      end
+      # all else - can't do anything
+    end
+  end
+
+  def current_size_and_limit
+    [identity_count, max_limit]
+  end
+
+  def logger
+    # we 'borrow' the codec's logger as we don't have our own
+    @base_codec.logger
+  end
+
+  def codec_without_usage_update(identity)
+    find_codec_value(identity).codec
+  end
+
+  def eviction_timestamp_for(identity)
+    find_codec_value(identity).eviction_timeout
+  end
+
+  private
+
+  def stream_codec(identity)
+    return base_codec if identity.nil?
+    record_codec_usage(identity) # returns codec
+  end
+
+  def find_codec_value(identity)
+    identity_map[identity]
+  end
+
+  # for nil stream this method is not called
+  def record_codec_usage(identity)
+    check_map_limits
+    # only start the cleaner if streams are in use
+    # continuous calls to start are OK
+    cleaner.start
+    auto_flusher.start
+    compo = find_codec_value(identity)
+    now = Time.now
+    compo.eviction_timeout = eviction_timestamp(now)
+    compo.auto_flush_timeout = auto_flush_timestamp(now)
+    compo.codec
+  end
+
+  def auto_flush_timestamp(now = Time.now)
+    now.to_f + @auto_flush_interval.to_f
+  end
+
+  def eviction_timestamp(now = Time.now)
+    now.to_i + @evict_timeout
+  end
+
+  def check_map_limits
+    UpperLimitReached.visit(self)
+    EightyPercentWarning.visit(self)
+  end
+
+  def codec_builder(hash, k)
+    codec = hash.empty? ? @base_codec : @base_codec.clone
+    codec.use_mapper_auto_flush if using_mapped_auto_flush?
+    compo = CodecValue.new(codec).tap do |o|
+      now = Time.now
+      o.eviction_timeout = eviction_timestamp(now)
+      o.auto_flush_timeout = auto_flush_timestamp(now)
+    end
+    hash.store(k, compo)
+  end
+
+  def no_streams?
+    identity_map.empty?
+  end
+
+  def using_mapped_auto_flush?
+    !@auto_flush_interval.nil?
+  end
+end end end