logstash-input-bro 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 98252e0b73f359a7d9970b1816199f19b7e4a9bc
+  data.tar.gz: 1fabda0d94eb4b9ba80f8f11940d7e6cba16120b
+SHA512:
+  metadata.gz: adaa8c0ee26c19d4cff2a3c5697464d027b61af6de91f319210249a0117a67ef9cb4159d61da1281c2c419e494e252c7cd88b26faa552921fb52c2475b81a6a5
+  data.tar.gz: 3ff95f51b92b1c82df5d5b33b113e4ef7e106141e285900f4dca78793059bec8277f5424e8b8c0abc19422c264fd5b5f95314d653c96f96860951a2e1ae1a713

data/CHANGELOG.md

@@ -0,0 +1,4 @@
+## 0.1.0
+ - Working with bro input files and multiple workers using mutex.
+## 0.0.1
+ - Snagged logstash-input-file code to use as base.

data/CONTRIBUTORS

@@ -0,0 +1,5 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Blake Mackey (brashendeavours)

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2015 BrashEndeavours <http://www.brashendeavours.co>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/NOTICE.TXT

@@ -0,0 +1,5 @@
+BrashEndeavours
+Copyright 2015 BrashEndeavours
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -0,0 +1,86 @@
+# Logstash Plugin
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+bin/plugin install --no-verify
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/inputs/bro.rb

@@ -0,0 +1,340 @@
+# encoding: utf-8
+require "logstash/namespace"
+require "logstash/inputs/base"
+require "logstash/codecs/identity_map_codec"
+
+require "pathname"
+require "socket" # for Socket.gethostname
+
+# Stream events from a bro log file, normally by tailing them in a manner
+# similar to `tail -0F` but optionally reading them from the
+# beginning.
+#
+# Each event is assumed to be one line, because of the nature of the
+# way bro logs are defined.
+#
+# The plugin aims to track changing files and emit new content as it's
+# appended to each file. It's not well-suited for reading a file from
+# beginning to end and storing all of it in a single event (not even
+# with the multiline codec or filter).
+#
+# ==== Tracking of current position in watched files
+#
+# The plugin keeps track of the current position in the file by
+# recording it in a separate file named sincedb. This makes it
+# possible to stop and restart Logstash and have it pick up where it
+# left off without missing the lines that were added to the file while
+# Logstash was stopped.
+#
+# By default, the sincedb file is placed in the home directory of the
+# user running Logstash with a filename based on the filename patterns
+# being watched (i.e. the `path` option). Thus, changing the filename
+# patterns will result in a new sincedb file being used and any
+# existing current position state will be lost. If you change your
+# patterns with any frequency it might make sense to explicitly choose
+# a sincedb path with the `sincedb_path` option.
+#
+# Sincedb files are text files with four columns:
+#
+# . The inode number (or equivalent).
+# . The major device number of the file system (or equivalent).
+# . The minor device number of the file system (or equivalent).
+# . The current byte offset within the file.
+#
+# On non-Windows systems you can obtain the inode number of a file
+# with e.g. `ls -li`.
+#
+# ==== File rotation
+#
+# File rotation is detected and handled by this input, regardless of
+# whether the file is rotated via a rename or a copy operation. To
+# support programs that write to the rotated file for some time after
+# the rotation has taken place, include both the original filename and
+# the rotated filename (e.g. /var/log/syslog and /var/log/syslog.1) in
+# the filename patterns to watch (the `path` option). Note that the
+# rotated filename will be treated as a new file so if
+# `start_position` is set to 'beginning' the rotated file will be
+# reprocessed.
+#
+# With the default value of `start_position` ('end') any messages
+# written to the end of the file between the last read operation prior
+# to the rotation and its reopening under the new name (an interval
+# determined by the `stat_interval` and `discover_interval` options)
+# will not get picked up.
+class LogStash::Inputs::Bro < LogStash::Inputs::Base
+  config_name "bro"
+
+  # The path to the file(s) to use as an input.
+  # You can use filename patterns here, such as `/var/log/*.log`.
+  # If you use a pattern like `/var/log/**/*.log`, a recursive search
+  # of `/var/log` will be done for all `*.log` files.
+  # Paths must be absolute and cannot be relative.
+  #
+  # You may also configure multiple paths. See an example
+  # on the <<array,Logstash configuration page>>.
+  config :path, :validate => :string, :required => true
+
+  # How often (in seconds) we stat files to see if they have been modified.
+  # Increasing this interval will decrease the number of system calls we make,
+  # but increase the time to detect new log lines.
+  config :stat_interval, :validate => :number, :default => 1
+
+  # How often (in seconds) we expand the filename patterns in the
+  # `path` option to discover new files to watch.
+  config :discover_interval, :validate => :number, :default => 15
+
+  # Path of the sincedb database file (keeps track of the current
+  # position of monitored log files) that will be written to disk.
+  # The default will write sincedb files to some path matching `$HOME/.sincedb*`
+  # NOTE: it must be a file path and not a directory path
+  config :sincedb_path, :validate => :string
+
+  # How often (in seconds) to write a since database with the current position of
+  # monitored log files.
+  config :sincedb_write_interval, :validate => :number, :default => 15
+
+  # Choose where Logstash starts initially reading files: at the beginning or
+  # at the end. The default behavior treats files like live streams and thus
+  # starts at the end. If you have old data you want to import, set this
+  # to 'beginning'.
+  #
+  # This option only modifies "first contact" situations where a file
+  # is new and not seen before, i.e. files that don't have a current
+  # position recorded in a sincedb file read by Logstash. If a file
+  # has already been seen before, this option has no effect and the
+  # position recorded in the sincedb file will be used.
+  config :start_position, :validate => [ "beginning", "end"], :default => "end"
+
+  # set the new line delimiter, defaults to "\n"
+  config :delimiter, :validate => :string, :default => "\n"
+
+  public
+  def register
+    require "addressable/uri"
+    require "filewatch/tail"
+    require "digest/md5"
+    @logger.info("Registering file input", :path => @path)
+    @host = Socket.gethostname.force_encoding(Encoding::UTF_8)
+
+    @tail_config = {
+      #:exclude => @exclude,
+      :stat_interval => @stat_interval,
+      :discover_interval => @discover_interval,
+      :sincedb_write_interval => @sincedb_write_interval,
+      :delimiter => @delimiter,
+      :logger => @logger,
+    }
+
+    if Pathname.new(path).relative?
+      raise ArgumentError.new("File paths must be absolute, relative path specified: #{path}")
+    end
+
+    if @sincedb_path.nil?
+      if ENV["SINCEDB_DIR"].nil? && ENV["HOME"].nil?
+        @logger.error("No SINCEDB_DIR or HOME environment variable set, I don't know where " \
+                      "to keep track of the files I'm watching. Either set " \
+                      "HOME or SINCEDB_DIR in your environment, or set sincedb_path in " \
+                      "in your Logstash config for the file input with " \
+                      "path '#{@path.inspect}'")
+        raise # TODO(sissel): HOW DO I FAIL PROPERLY YO
+      end
+
+      #pick SINCEDB_DIR if available, otherwise use HOME
+      sincedb_dir = ENV["SINCEDB_DIR"] || ENV["HOME"]
+
+      # Join by ',' to make it easy for folks to know their own sincedb
+      # generated path (vs, say, inspecting the @path array)
+      @sincedb_path = File.join(sincedb_dir, ".sincedb_" + Digest::MD5.hexdigest(@path))
+
+      # Migrate any old .sincedb to the new file (this is for version <=1.1.1 compatibility)
+      old_sincedb = File.join(sincedb_dir, ".sincedb")
+      if File.exists?(old_sincedb)
+        @logger.info("Renaming old ~/.sincedb to new one", :old => old_sincedb,
+                     :new => @sincedb_path)
+        File.rename(old_sincedb, @sincedb_path)
+      end
+
+      @logger.info("No sincedb_path set, generating one based on the file path",
+                   :sincedb_path => @sincedb_path, :path => @path)
+    end
+
+    if File.directory?(@sincedb_path)
+      raise ArgumentError.new("The \"sincedb_path\" argument must point to a file, received a directory: \"#{@sincedb_path}\"")
+    end
+
+    @tail_config[:sincedb_path] = @sincedb_path
+
+    if @start_position == "beginning"
+      @tail_config[:start_new_files_at] = :beginning
+    end
+
+    @codec = LogStash::Codecs::IdentityMapCodec.new(LogStash::Codecs::Plain.new)
+    @filter_initialized = false
+    @mutex = Mutex.new
+  end # def register
+
+  def begin_tailing
+    stop # if the pipeline restarts this input.
+    @tail = FileWatch::Tail.new(@tail_config)
+    @tail.logger = @logger
+    @tail.tail(path)
+  end
+
+  def run(queue)
+    begin_tailing
+    @tail.subscribe do |path, line|
+      log_line_received(path, line)
+      @codec.decode(line, path) do |event|
+        # path is the identity
+        # Note: this block is cached in the
+        # identity_map_codec for use when
+        # buffered lines are flushed.
+        
+        # Ignore the setup lines initially
+        if event["message"].start_with?('#')
+          next
+        end
+
+        queue << add_path_meta(event, path)
+      end
+    end
+  end # def run
+
+  def log_line_received(path, line)
+    return if !@logger.debug?
+    @logger.debug("Received line", :path => path, :text => line)
+  end
+
+  def add_path_meta(event, path)
+    @path = path      
+    # Until we get a line that doesn't have a comment, then we 
+    # check if we're setup.
+    unless @filter_initialized
+      initialize_filter()
+      print_config()
+    end
+    event["@host"] = @host
+    event["@path"] = @path
+    event["@message"] = event["message"]
+    event.remove("message")
+    event.remove("host") unless event["host"].nil?
+    event.remove("@version") unless event["@version"].nil?
+
+    values = event["@message"].split(/#{@separator}/)
+    values.each_index do |i|
+
+      if values[i].start_with?(@empty_field, @unset_field) then next end
+
+      field_name = @fields[i] || "uninitialized#{i+1}"
+      field_type = @types[i] || "string"
+
+
+      if field_type.start_with?("interval", "double")
+        values[i] = values[i].to_f
+      elsif field_type.start_with?("count", "int")
+        values[i] = values[i].to_i
+      elsif field_type.start_with?("set", "vector")
+        if field_type =~ /interval/ || /double/
+          values[i] = values[i].split(',').map(&:to_f)      
+        elsif field_type =~ /int/ || /count/
+          values[i] = values[i].split(',').map(&:to_i)
+        elsif field_type =~ /time/
+          values[i] = values[i].split(',').map do |block_value|
+            # Truncate timestamp to millisecond precision
+            secs = BigDecimal.new(block_value)
+            msec  = secs * 1000 # convert to whole number of milliseconds
+            msec  = msec.to_i
+            block_value = Time.at(msec / 1000, (msec % 1000) * 1000).utc
+          end
+        else
+          values[i] = values[i].split(',')
+        end
+          
+      elsif field_type.start_with?("time") # Create an actual timestamp
+        # Truncate timestamp to millisecond precision
+        secs = BigDecimal.new(values[i])
+        #event["#{field_name}_secs"] = secs.to_f
+        msec  = secs * 1000 # convert to whole number of milliseconds
+        msec  = msec.to_i
+        values[i] = Time.at(msec / 1000, (msec % 1000) * 1000).utc
+      end
+
+      field_array = field_name.split('.')
+      field_hash = field_array.reverse.inject(values[i]) { |a, n| { n => a } }
+      field_hash = field_hash[field_hash.keys[0]]
+      #event[field_name] = values[i]
+      if event.include?(field_array[0])
+        event[field_array[0]] = event[field_array[0]].to_hash.merge!(field_hash) { |_, v1, v2| [v1,v2] }
+      else
+        event[field_array[0]] = field_hash
+      end
+    end
+
+    # Add some additional data
+    if event.include?("@timestamp")
+      event["timestamp"]               = {}
+      event["timestamp"]["start"]      = event["@timestamp"]
+      if event.include?("duration")
+        event["timestamp"]["duration"] = event["duration"]
+        event["timestamp"]["end"]      = LogStash::Timestamp.new(event["timestamp"]["start"] + event["duration"].to_f) 
+        event.remove("duration")
+      end
+    end
+    #event["bro_logtype"] = @meta[current_event][:logname]
+    #rescue => e
+    # event.tag "_broparsefailure"
+    # @logger.warn("Trouble parsing bro", :event => event, :exception => e)
+    # print e
+    # return
+    #end # begin
+
+
+    decorate(event)
+    event
+  end
+
+  def initialize_filter()
+    @mutex.synchronize do
+      unless @filter_initialized
+        lines = File.foreach(@path).first(8)
+        lines.each do |line|
+          startword = line.chomp!.split.first
+          case startword
+          when "#separator"
+            @separator     = line.partition(/ /)[2]
+          when "#set_separator"
+            @set_separator = line.partition(/#{@separator}/)[2]
+          when "#empty_field"
+            @empty_field   = line.partition(/#{@separator}/)[2]
+          when "#unset_field"
+            @unset_field   = line.partition(/#{@separator}/)[2]
+          when "#path"
+            @logname       = line.partition(/#{@separator}/)[2]
+          when "#fields"
+            @fields        = line.partition(/#{@separator}/)[2].split(/#{@separator}/)
+          when "#types"
+            @types         = line.partition(/#{@separator}/)[2].split(/#{@separator}/)
+          end
+        end # line.each
+        @filter_initialized = true
+      end # filter_initialized
+    end # synchronize
+  end # def initialize_filter
+  
+  def print_config()
+    @logger.info("separator:      \"#{@separator}\"")
+    @logger.info("set separator:  \"#{@set_separator}\"")
+    @logger.info("empty field:    \"#{@empty_field}\"")
+    @logger.info("unset field:    \"#{@unset_field}\"")
+    @logger.info("logname:        \"#{@logname}\"")
+    @logger.info("columns:        \"#{@fields}\"")
+    @logger.info("types:          \"#{@types}\"")
+  end # def print_path_config
+
+  def stop
+    # in filewatch >= 0.6.7, quit will closes and forget all files
+    # but it will write their last read positions to since_db
+    # beforehand
+    @tail.quit if @tail
+  end
+end # class LogStash::Inputs::File

data/logstash-input-bro.gemspec

@@ -0,0 +1,34 @@
+Gem::Specification.new do |s|
+
+  s.name            = 'logstash-input-bro'
+  s.version         = '0.1.0'
+  s.licenses        = ['Apache License (2.0)']
+  s.summary         = "Stream events from bro formatted files."
+  s.description     = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
+  s.authors = ["Blake Mackey"]
+  s.email = 'blake_mackey@hotmail.com'
+  s.homepage = "http://github.com/brashendeavours/logstash-input-bro"
+  s.require_paths = ["lib"]
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
+
+  # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "input" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core", ">= 2.0.0.beta2", "< 3.0.0"
+
+  s.add_runtime_dependency 'logstash-codec-plain'
+  s.add_runtime_dependency 'addressable'
+  s.add_runtime_dependency 'filewatch', ['>= 0.6.7', '~> 0.6']
+  s.add_runtime_dependency 'logstash-codec-multiline', ['~> 2.0.3']
+
+  s.add_development_dependency 'stud', ['~> 0.0.19']
+  s.add_development_dependency 'logstash-devutils'
+  s.add_development_dependency 'logstash-codec-json'
+end
+

data/spec/inputs/bro_spec.rb

@@ -0,0 +1,307 @@
+# encoding: utf-8
+
+require "logstash/devutils/rspec/spec_helper"
+require "tempfile"
+require "stud/temporary"
+require "logstash/inputs/bro"
+
+FILE_DELIMITER = LogStash::Environment.windows? ? "\r\n" : "\n"
+
+describe LogStash::Inputs::Bro do
+
+  before(:all) do
+    @abort_on_exception = Thread.abort_on_exception
+    Thread.abort_on_exception = true
+  end
+
+  after(:all) do
+    Thread.abort_on_exception = @abort_on_exception
+  end
+
+  it_behaves_like "an interruptible input plugin" do
+    let(:config) do
+      {
+        "path" => Stud::Temporary.pathname,
+        "sincedb_path" => Stud::Temporary.pathname
+      }
+    end
+  end
+
+  it "should starts at the end of an existing file" do
+    tmpfile_path = Stud::Temporary.pathname
+    sincedb_path = Stud::Temporary.pathname
+
+    conf = <<-CONFIG
+      input {
+        bro {
+          type => "blah"
+          path => "#{tmpfile_path}"
+          sincedb_path => "#{sincedb_path}"
+          delimiter => "#{FILE_DELIMITER}"
+        }
+      }
+    CONFIG
+
+    File.open(tmpfile_path, "w") do |fd|
+      fd.puts("ignore me 1")
+      fd.puts("ignore me 2")
+    end
+
+    events = input(conf) do |pipeline, queue|
+
+      # at this point the plugins
+      # threads might still be initializing so we cannot know when the
+      # file plugin will have seen the original file, it could see it
+      # after the first(s) hello world appends below, hence the
+      # retry logic.
+
+      events = []
+
+      retries = 0
+      while retries < 20
+        File.open(tmpfile_path, "a") do |fd|
+          fd.puts("hello")
+          fd.puts("world")
+        end
+
+        if queue.size >= 2
+          events = 2.times.collect { queue.pop }
+          break
+        end
+
+        sleep(0.1)
+        retries += 1
+      end
+
+      events
+    end
+
+    insist { events[0]["message"] } == "hello"
+    insist { events[1]["message"] } == "world"
+  end
+
+  it "should start at the beginning of an existing file" do
+    tmpfile_path = Stud::Temporary.pathname
+    sincedb_path = Stud::Temporary.pathname
+
+    conf = <<-CONFIG
+      input {
+        bro {
+          type => "blah"
+          path => "#{tmpfile_path}"
+          start_position => "beginning"
+          sincedb_path => "#{sincedb_path}"
+          delimiter => "#{FILE_DELIMITER}"
+        }
+      }
+    CONFIG
+
+    File.open(tmpfile_path, "a") do |fd|
+      fd.puts("hello")
+      fd.puts("world")
+    end
+
+    events = input(conf) do |pipeline, queue|
+      2.times.collect { queue.pop }
+    end
+
+    insist { events[0]["message"] } == "hello"
+    insist { events[1]["message"] } == "world"
+  end
+
+  it "should restarts at the sincedb value" do
+    tmpfile_path = Stud::Temporary.pathname
+    sincedb_path = Stud::Temporary.pathname
+
+    conf = <<-CONFIG
+      input {
+        bro {
+          type => "blah"
+          path => "#{tmpfile_path}"
+          start_position => "beginning"
+          sincedb_path => "#{sincedb_path}"
+          delimiter => "#{FILE_DELIMITER}"
+        }
+      }
+    CONFIG
+
+    File.open(tmpfile_path, "w") do |fd|
+      fd.puts("hello3")
+      fd.puts("world3")
+    end
+
+    events = input(conf) do |pipeline, queue|
+      2.times.collect { queue.pop }
+    end
+
+    insist { events[0]["message"] } == "hello3"
+    insist { events[1]["message"] } == "world3"
+
+    File.open(tmpfile_path, "a") do |fd|
+      fd.puts("foo")
+      fd.puts("bar")
+      fd.puts("baz")
+    end
+
+    events = input(conf) do |pipeline, queue|
+      3.times.collect { queue.pop }
+    end
+
+    insist { events[0]["message"] } == "foo"
+    insist { events[1]["message"] } == "bar"
+    insist { events[2]["message"] } == "baz"
+  end
+
+  it "should not overwrite existing path and host fields" do
+    tmpfile_path = Stud::Temporary.pathname
+    sincedb_path = Stud::Temporary.pathname
+
+    conf = <<-CONFIG
+      input {
+        bro {
+          type => "blah"
+          path => "#{tmpfile_path}"
+          start_position => "beginning"
+          sincedb_path => "#{sincedb_path}"
+          delimiter => "#{FILE_DELIMITER}"
+          codec => "json"
+        }
+      }
+    CONFIG
+
+    File.open(tmpfile_path, "w") do |fd|
+      fd.puts('{"path": "my_path", "host": "my_host"}')
+      fd.puts('{"my_field": "my_val"}')
+    end
+
+    events = input(conf) do |pipeline, queue|
+      2.times.collect { queue.pop }
+    end
+
+    insist { events[0]["path"] } == "my_path"
+    insist { events[0]["host"] } == "my_host"
+
+    insist { events[1]["path"] } == "#{tmpfile_path}"
+    insist { events[1]["host"] } == "#{Socket.gethostname.force_encoding(Encoding::UTF_8)}"
+  end
+
+  context "when sincedb_path is an existing directory" do
+    let(:tmpfile_path) { Stud::Temporary.pathname }
+    let(:sincedb_path) { Stud::Temporary.directory }
+    subject { LogStash::Inputs::File.new("path" => tmpfile_path, "sincedb_path" => sincedb_path) }
+
+    after :each do
+      FileUtils.rm_rf(sincedb_path)
+    end
+
+    it "should raise exception" do
+      expect { subject.register }.to raise_error(ArgumentError)
+    end
+  end
+
+  context "when #run is called multiple times", :unix => true do
+    let(:tmpdir_path)  { Stud::Temporary.directory }
+    let(:sincedb_path) { Stud::Temporary.pathname }
+    let(:file_path)    { "#{tmpdir_path}/a.log" }
+    let(:buffer)       { [] }
+    let(:lsof)         { [] }
+    let(:stop_proc) do
+      lambda do |input, arr|
+        Thread.new(input, arr) do |i, a|
+          sleep 0.5
+          a << `lsof -p #{Process.pid} | grep "a.log"`
+          i.stop
+        end
+      end
+    end
+
+    subject { LogStash::Inputs::File.new("path" => tmpdir_path + "/*.log", "start_position" => "beginning", "sincedb_path" => sincedb_path) }
+
+    after :each do
+      FileUtils.rm_rf(tmpdir_path)
+      FileUtils.rm_rf(sincedb_path)
+    end
+    before do
+      File.open(file_path, "w") do |fd|
+        fd.puts('foo')
+        fd.puts('bar')
+      end
+    end
+    it "should only have one set of files open" do
+      subject.register
+      lsof_before = `lsof -p #{Process.pid} | grep #{file_path}`
+      expect(lsof_before).to eq("")
+      stop_proc.call(subject, lsof)
+      subject.run(buffer)
+      expect(lsof.first).not_to eq("")
+      stop_proc.call(subject, lsof)
+      subject.run(buffer)
+      expect(lsof.last).to eq(lsof.first)
+    end
+  end
+
+  context "when wildcard path and a multiline codec is specified" do
+    let(:tmpdir_path)  { Stud::Temporary.directory }
+    let(:sincedb_path) { Stud::Temporary.pathname }
+    let(:conf) do
+      <<-CONFIG
+        input {
+          bro {
+            type => "blah"
+            path => "#{tmpdir_path}/*.log"
+            start_position => "beginning"
+            sincedb_path => "#{sincedb_path}"
+            delimiter => "#{FILE_DELIMITER}"
+            codec => multiline { pattern => "^\s" what => previous }
+          }
+        }
+      CONFIG
+    end
+
+    let(:writer_proc) do
+      -> do
+        File.open("#{tmpdir_path}/a.log", "a") do |fd|
+          fd.puts("line1.1-of-a")
+          fd.puts("  line1.2-of-a")
+          fd.puts("  line1.3-of-a")
+          fd.puts("line2.1-of-a")
+        end
+
+        File.open("#{tmpdir_path}/z.log", "a") do |fd|
+          fd.puts("line1.1-of-z")
+          fd.puts("  line1.2-of-z")
+          fd.puts("  line1.3-of-z")
+          fd.puts("line2.1-of-z")
+        end
+      end
+    end
+
+    after do
+      FileUtils.rm_rf(tmpdir_path)
+    end
+
+    let(:event_count) { 2 }
+
+    it "collects separate multiple line events from each file" do
+      writer_proc.call
+
+      events = input(conf) do |pipeline, queue|
+        queue.size.times.collect { queue.pop }
+      end
+
+      expect(events.size).to eq(event_count)
+
+      e1_message = events[0]["message"]
+      e2_message = events[1]["message"]
+
+      # can't assume File A will be read first
+      if e1_message.start_with?('line1.1-of-z')
+        expect(e1_message).to eq("line1.1-of-z#{FILE_DELIMITER}  line1.2-of-z#{FILE_DELIMITER}  line1.3-of-z")
+        expect(e2_message).to eq("line1.1-of-a#{FILE_DELIMITER}  line1.2-of-a#{FILE_DELIMITER}  line1.3-of-a")
+      else
+        expect(e1_message).to eq("line1.1-of-a#{FILE_DELIMITER}  line1.2-of-a#{FILE_DELIMITER}  line1.3-of-a")
+        expect(e2_message).to eq("line1.1-of-z#{FILE_DELIMITER}  line1.2-of-z#{FILE_DELIMITER}  line1.3-of-z")
+      end
+    end
+  end
+end

metadata

@@ -0,0 +1,179 @@
+--- !ruby/object:Gem::Specification
+name: logstash-input-bro
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Blake Mackey
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2015-12-28 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 2.0.0.beta2
+    - - <
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+  name: logstash-core
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 2.0.0.beta2
+    - - <
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-codec-plain
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: addressable
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 0.6.7
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '0.6'
+  name: filewatch
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 0.6.7
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '0.6'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 2.0.3
+  name: logstash-codec-multiline
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 2.0.3
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.0.19
+  name: stud
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.0.19
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-codec-json
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program
+email: blake_mackey@hotmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CONTRIBUTORS
+- Gemfile
+- LICENSE
+- NOTICE.TXT
+- README.md
+- lib/logstash/inputs/bro.rb
+- logstash-input-bro.gemspec
+- spec/inputs/bro_spec.rb
+homepage: http://github.com/brashendeavours/logstash-input-bro
+licenses:
+- Apache License (2.0)
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: input
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.4.5
+signing_key:
+specification_version: 4
+summary: Stream events from bro formatted files.
+test_files:
+- spec/inputs/bro_spec.rb