logstash-input-file 4.1.3 → 4.1.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 44f11a07375e6cf964220179c9cea4259a9bb6829dbbdda85e05fef737a9e214
-  data.tar.gz: 9005ada3317a3d947bce138f9bc014647f79ad871db7cfd373642f51f94ca8bc
+  metadata.gz: e7cb22ef0489b15ab92212c179444aefcd5be9e95fc4984c28f83de3397bf54c
+  data.tar.gz: 7a68f15af95595390a4f44e3fd9f641522e1e8ce3feafb7b299b1e056b5b2df7
 SHA512:
-  metadata.gz: 772fedf54c74b08d660f2a6aca1dd43f9ab04cb9290ef67d409e26419931bcf0fa26de027c075d561f35f40d55ac8b43c376c7b7c88f1f7982f1eb243361c7a3
-  data.tar.gz: dc5dc4aa91b870368967f92fc4129faa4e2b470e0bd942dda965711fab90def3658e9efd8f9158f3f98193a5aedcf23322bf65c7f5a28a4c32b84d6aac370118
+  metadata.gz: fedb59ccffbf65f45f1f88dc76cd5e45d126866f75e4eb9271ef44a1e2b7922f98ff49643272cfdfa5f1ac01d445104f8a6c308f71e8255e85cc19c95809122a
+  data.tar.gz: d10e33bf905addfc71a5ce4ce026f3fe4b7f80871b1ba087f41fc3ed7b7981289986ec61a5b4b96ef9bd37af4b26ace1219b99c723032a49a1779ee9e52010f2

data/CHANGELOG.md

@@ -1,3 +1,13 @@
+## 4.1.4
+  - Fixed a regression where files discovered after first discovery were not
+    always read from the beginning. Applies to tail mode only.
+    [#198](https://github.com/logstash-plugins/logstash-input-file/issues/198)
+  - Added much better support for file rotation schemes of copy/truncate and
+    rename cascading. Applies to tail mode only.
+  - Added support for processing files over remote mounts e.g. NFS. Before, it
+    was possible to read into memory allocated but not filled with data resulting
+    in ASCII NUL (0) bytes in the message field. Now, files are read up to the
+    size as given by the remote filesystem client. Applies to tail and read modes.
 ## 4.1.3
   - Fixed `read` mode of regular files sincedb write is requested in each read loop
     iteration rather than waiting for the end-of-file to be reached. Note: for gz files,

data/JAR_VERSION

@@ -1 +1 @@
-1.0.0
+1.0.1

data/README.md

@@ -2,9 +2,6 @@
 Travis Build
 [![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-input-file.svg)](https://travis-ci.org/logstash-plugins/logstash-input-file)
 
-Jenkins Build
-[![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-input-file.svg)](https://travis-ci.org/logstash-plugins/logstash-input-file)
-
 This is a plugin for [Logstash](https://github.com/elastic/logstash).
 
 It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.

data/docs/index.asciidoc

@@ -78,12 +78,6 @@ Read mode also allows for an action to take place after processing the file comp
 In the past attempts to simulate a Read mode while still assuming infinite streams
 was not ideal and a dedicated Read mode is an improvement.
 
-==== Reading from remote network volumes
-
-The file input is not tested on remote filesystems such as NFS, Samba, s3fs-fuse, etc. These
-remote filesystems typically have behaviors that are very different from local filesystems and
-are therefore unlikely to work correctly when used with the file input.
-
 ==== Tracking of current position in watched files
 
 The plugin keeps track of the current position in each file by
@@ -103,6 +97,10 @@ A different `sincedb_path` must be used for each input. Using the same
 path will cause issues. The read checkpoints for each input must be
 stored in a different path so the information does not override.
 
+Files are tracked via an identifier. This identifier is made up of the
+inode, major device number and minor device number. In windows, a different
+identifier is taken from a `kernel32` API call.
+
 Sincedb records can now be expired meaning that read positions of older files
 will not be remembered after a certain time period. File systems may need to reuse
 inodes for new content. Ideally, we would not use the read position of old content,
@@ -123,6 +121,19 @@ old sincedb records converted to the new format, this is blank.
 On non-Windows systems you can obtain the inode number of a file
 with e.g. `ls -li`.
 
+==== Reading from remote network volumes
+
+The file input is not thoroughly tested on remote filesystems such as NFS,
+Samba, s3fs-fuse, etc, however NFS is occasionally tested. The file size as given by
+the remote FS client is used to govern how much data to read at any given time to
+prevent reading into allocated but yet unfilled memory.
+As we use the device major and minor in the identifier to track "last read"
+positions of files and on remount the device major and minor can change, the
+sincedb records may not match across remounts.
+Read mode might not be suitable for remote filesystems as the file size at
+discovery on the client side may not be the same as the file size on the remote side
+due to latency in the remote to client copy process.
+
 ==== File rotation in Tail mode
 
 File rotation is detected and handled by this input, regardless of
@@ -130,16 +141,15 @@ whether the file is rotated via a rename or a copy operation. To
 support programs that write to the rotated file for some time after
 the rotation has taken place, include both the original filename and
 the rotated filename (e.g. /var/log/syslog and /var/log/syslog.1) in
-the filename patterns to watch (the `path` option). Note that the
-rotated filename will be treated as a new file so if
-`start_position` is set to 'beginning' the rotated file will be
-reprocessed.
-
-With the default value of `start_position` ('end') any messages
-written to the end of the file between the last read operation prior
-to the rotation and its reopening under the new name (an interval
-determined by the `stat_interval` and `discover_interval` options)
-will not get picked up.
+the filename patterns to watch (the `path` option).
+For a rename, the inode will be detected as having moved from
+`/var/log/syslog` to `/var/log/syslog.1` and so the "state" is moved
+internally too, the old content will not be reread but any new content
+on the renamed file will be read.
+For copy/truncate the copied content into a new file path, if discovered, will
+be treated as a new discovery and be read from the beginning. The copied file
+paths should therefore not be in the filename patterns to watch (the `path` option).
+The truncation will be detected and the "last read" position updated to zero.
 
 [id="plugins-{type}s-{plugin}-options"]
 ==== File Input Configuration Options

data/lib/filewatch/bootstrap.rb

@@ -1,7 +1,5 @@
 # encoding: utf-8
-require "rbconfig"
 require "pathname"
-# require "logstash/environment"
 
 ## Common setup
 #  all the required constants and files
@@ -13,36 +11,26 @@ module FileWatch
   # this is used in the read loop e.g.
   # @opts[:file_chunk_count].times do
   # where file_chunk_count defaults to this constant
-  FIXNUM_MAX = (2**(0.size * 8 - 2) - 1)
+  MAX_ITERATIONS = (2**(0.size * 8 - 2) - 2) / 32768
 
   require_relative "helper"
 
-  module WindowsInode
-    def prepare_inode(path, stat)
-      fileId = Winhelper.GetWindowsUniqueFileIdentifier(path)
-      [fileId, 0, 0] # dev_* doesn't make sense on Windows
-    end
-  end
-
-  module UnixInode
-    def prepare_inode(path, stat)
-      [stat.ino.to_s, stat.dev_major, stat.dev_minor]
-    end
-  end
-
-  jar_version = Pathname.new(__FILE__).dirname.join("../../JAR_VERSION").realpath.read.strip
-
+  gem_root_dir = Pathname.new(__FILE__).dirname.join("../../").realpath
+  jar_version = gem_root_dir.join("JAR_VERSION").read.strip
+  fullpath = gem_root_dir.join("lib/jars/filewatch-#{jar_version}.jar").expand_path.to_path
   require "java"
-  require_relative "../../lib/jars/filewatch-#{jar_version}.jar"
+  require fullpath
   require "jruby_file_watch"
 
   if LogStash::Environment.windows?
     require_relative "winhelper"
+    require_relative "stat/windows_path"
+    PathStatClass = Stat::WindowsPath
     FileOpener = FileExt
-    InodeMixin = WindowsInode
   else
+    require_relative "stat/generic"
+    PathStatClass = Stat::Generic
     FileOpener = ::File
-    InodeMixin = UnixInode
   end
 
   # Structs can be used as hash keys because they compare by value
@@ -54,6 +42,7 @@ module FileWatch
   end
 
   BufferExtractResult = Struct.new(:lines, :warning, :additional)
+  LoopControlResult = Struct.new(:count, :size, :more)
 
   class NoSinceDBPathGiven < StandardError; end
 

data/lib/filewatch/discoverer.rb

@@ -10,8 +10,8 @@ module FileWatch
     include LogStash::Util::Loggable
 
     def initialize(watched_files_collection, sincedb_collection, settings)
-      @watching = []
-      @exclude = []
+      @watching = Concurrent::Array.new
+      @exclude = Concurrent::Array.new
       @watched_files_collection = watched_files_collection
       @sincedb_collection = sincedb_collection
       @settings = settings
@@ -21,13 +21,13 @@ module FileWatch
     def add_path(path)
       return if @watching.member?(path)
       @watching << path
-      discover_files(path)
+      discover_files_new_path(path)
       self
     end
 
     def discover
       @watching.each do |path|
-        discover_files(path)
+        discover_files_ongoing(path)
       end
     end
 
@@ -37,7 +37,7 @@ module FileWatch
       @exclude.each do |pattern|
         if watched_file.pathname.fnmatch?(pattern)
           if new_discovery
-            logger.debug("Discoverer can_exclude?: #{watched_file.path}: skipping " +
+            logger.trace("Discoverer can_exclude?: #{watched_file.path}: skipping " +
               "because it matches exclude #{pattern}")
           end
           watched_file.unwatch
@@ -47,45 +47,52 @@ module FileWatch
       false
     end
 
-    def discover_files(path)
-      globbed = Dir.glob(path)
-      globbed = [path] if globbed.empty?
-      logger.debug("Discoverer found files, count: #{globbed.size}")
-      globbed.each do |file|
-        logger.debug("Discoverer found file, path: #{file}")
+    def discover_files_new_path(path)
+      discover_any_files(path, false)
+    end
+
+    def discover_files_ongoing(path)
+      discover_any_files(path, true)
+    end
+
+    def discover_any_files(path, ongoing)
+      fileset = Dir.glob(path).select{|f| File.file?(f) && !File.symlink?(f)}
+      logger.trace("discover_files",  "count" => fileset.size)
+      fileset.each do |file|
         pathname = Pathname.new(file)
-        next unless pathname.file?
-        next if pathname.symlink?
         new_discovery = false
         watched_file = @watched_files_collection.watched_file_by_path(file)
         if watched_file.nil?
-          logger.debug("Discoverer discover_files: #{path}: new: #{file} (exclude is #{@exclude.inspect})")
           new_discovery = true
-          watched_file = WatchedFile.new(pathname, pathname.stat, @settings)
+          watched_file = WatchedFile.new(pathname, PathStatClass.new(pathname), @settings)
         end
         # if it already unwatched or its excluded then we can skip
         next if watched_file.unwatched? || can_exclude?(watched_file, new_discovery)
 
+        logger.trace("discover_files handling:", "new discovery"=> new_discovery, "watched_file details" => watched_file.details)
+
         if new_discovery
-          if watched_file.file_ignorable?
-            logger.debug("Discoverer discover_files: #{file}: skipping because it was last modified more than #{@settings.ignore_older} seconds ago")
-            # on discovery ignorable watched_files are put into the ignored state and that
-            # updates the size from the internal stat
-            # so the existing contents are not read.
-            # because, normally, a newly discovered file will
-            # have a watched_file size of zero
-            # they are still added to the collection so we know they are there for the next periodic discovery
-            watched_file.ignore
-          end
-          # now add the discovered file to the watched_files collection and adjust the sincedb collections
-          @watched_files_collection.add(watched_file)
+          watched_file.initial_completed if ongoing
           # initially when the sincedb collection is filled with records from the persistence file
           # each value is not associated with a watched file
           # a sincedb_value can be:
           #   unassociated
           #   associated with this watched_file
           #   associated with a different watched_file
-          @sincedb_collection.associate(watched_file)
+          if @sincedb_collection.associate(watched_file)
+            if watched_file.file_ignorable?
+              logger.trace("Discoverer discover_files: #{file}: skipping because it was last modified more than #{@settings.ignore_older} seconds ago")
+              # on discovery ignorable watched_files are put into the ignored state and that
+              # updates the size from the internal stat
+              # so the existing contents are not read.
+              # because, normally, a newly discovered file will
+              # have a watched_file size of zero
+              # they are still added to the collection so we know they are there for the next periodic discovery
+              watched_file.ignore_as_unread
+            end
+            # now add the discovered file to the watched_files collection and adjust the sincedb collections
+            @watched_files_collection.add(watched_file)
+          end
         end
         # at this point the watched file is created, is in the db but not yet opened or being processed
       end

data/lib/filewatch/observing_base.rb

@@ -44,7 +44,8 @@ module FileWatch
         :exclude => [],
         :start_new_files_at => :end,
         :delimiter => "\n",
-        :file_chunk_count => FIXNUM_MAX,
+        :file_chunk_count => MAX_ITERATIONS,
+        :file_chunk_size => FILE_READ_SIZE,
         :file_sort_by => "last_modified",
         :file_sort_direction => "asc",
       }.merge(opts)

data/lib/filewatch/read_mode/handlers/base.rb

@@ -19,7 +19,7 @@ module FileWatch module ReadMode module Handlers
     end
 
     def handle(watched_file)
-      logger.debug("handling: #{watched_file.path}")
+      logger.trace("handling: #{watched_file.path}")
       unless watched_file.has_listener?
         watched_file.set_listener(@observer)
       end
@@ -34,7 +34,7 @@ module FileWatch module ReadMode module Handlers
 
     def open_file(watched_file)
       return true if watched_file.file_open?
-      logger.debug("opening #{watched_file.path}")
+      logger.trace("opening #{watched_file.path}")
       begin
         watched_file.open
       rescue
@@ -46,7 +46,7 @@ module FileWatch module ReadMode module Handlers
           logger.warn("failed to open #{watched_file.path}: #{$!.inspect}, #{$!.backtrace.take(3)}")
           watched_file.last_open_warning_at = now
         else
-          logger.debug("suppressed warning for `failed to open` #{watched_file.path}: #{$!.inspect}")
+          logger.trace("suppressed warning for `failed to open` #{watched_file.path}: #{$!.inspect}")
         end
         watched_file.watch # set it back to watch so we can try it again
       end
@@ -65,13 +65,26 @@ module FileWatch module ReadMode module Handlers
       elsif sincedb_value.watched_file == watched_file
         update_existing_sincedb_collection_value(watched_file, sincedb_value)
       else
-        logger.warn? && logger.warn("mismatch on sincedb_value.watched_file, this should have been handled by Discoverer")
+        msg = "add_or_update_sincedb_collection: the found sincedb_value has a watched_file - this is a rename, switching inode to this watched file"
+        logger.trace(msg)
+        existing_watched_file = sincedb_value.watched_file
+        if existing_watched_file.nil?
+          sincedb_value.set_watched_file(watched_file)
+          logger.trace("add_or_update_sincedb_collection: switching as new file")
+          watched_file.rotate_as_file
+          watched_file.update_bytes_read(sincedb_value.position)
+        else
+          sincedb_value.set_watched_file(watched_file)
+          logger.trace("add_or_update_sincedb_collection: switching from...", "watched_file details" => watched_file.details)
+          watched_file.rotate_from(existing_watched_file)
+        end
+
       end
       watched_file.initial_completed
     end
 
     def update_existing_sincedb_collection_value(watched_file, sincedb_value)
-      logger.debug("update_existing_sincedb_collection_value: #{watched_file.path}, last value #{sincedb_value.position}, cur size #{watched_file.last_stat_size}")
+      logger.trace("update_existing_sincedb_collection_value: #{watched_file.path}, last value #{sincedb_value.position}, cur size #{watched_file.last_stat_size}")
       # sincedb_value is the source of truth
       watched_file.update_bytes_read(sincedb_value.position)
     end
@@ -79,7 +92,7 @@ module FileWatch module ReadMode module Handlers
     def add_new_value_sincedb_collection(watched_file)
       sincedb_value = SincedbValue.new(0)
       sincedb_value.set_watched_file(watched_file)
-      logger.debug("add_new_value_sincedb_collection: #{watched_file.path}", "position" => sincedb_value.position)
+      logger.trace("add_new_value_sincedb_collection: #{watched_file.path}", "position" => sincedb_value.position)
       sincedb_collection.set(watched_file.sincedb_key, sincedb_value)
     end
   end

data/lib/filewatch/read_mode/handlers/read_file.rb

@@ -5,40 +5,51 @@ module FileWatch module ReadMode module Handlers
     def handle_specifically(watched_file)
       if open_file(watched_file)
         add_or_update_sincedb_collection(watched_file) unless sincedb_collection.member?(watched_file.sincedb_key)
-        @settings.file_chunk_count.times do
+        loop do
           break if quit?
-          begin
-            data = watched_file.file_read(@settings.file_chunk_size)
-            result = watched_file.buffer_extract(data) # expect BufferExtractResult
-            logger.info(result.warning, result.additional) unless result.warning.empty?
-            result.lines.each do |line|
-              watched_file.listener.accept(line)
-              # sincedb position is independent from the watched_file bytes_read
-              sincedb_collection.increment(watched_file.sincedb_key, line.bytesize + @settings.delimiter_byte_size)
-            end
-            # instead of tracking the bytes_read line by line we need to track by the data read size.
-            # because we initially seek to the bytes_read not the sincedb position
-            watched_file.increment_bytes_read(data.bytesize)
-            sincedb_collection.request_disk_flush
-          rescue EOFError
-            # flush the buffer now in case there is no final delimiter
-            line = watched_file.buffer.flush
-            watched_file.listener.accept(line) unless line.empty?
-            watched_file.listener.eof
-            watched_file.file_close
-            # unset_watched_file will set sincedb_value.position to be watched_file.bytes_read
-            sincedb_collection.unset_watched_file(watched_file)
-            watched_file.listener.deleted
-            watched_file.unwatch
-            break
-          rescue Errno::EWOULDBLOCK, Errno::EINTR
-            watched_file.listener.error
-            break
-          rescue => e
-            logger.error("read_to_eof: general error reading #{watched_file.path} - error: #{e.inspect}")
-            watched_file.listener.error
-            break
+          loop_control = watched_file.loop_control_adjusted_for_stat_size
+          controlled_read(watched_file, loop_control)
+          sincedb_collection.request_disk_flush
+          break unless loop_control.more
+        end
+        if watched_file.all_read?
+          # flush the buffer now in case there is no final delimiter
+          line = watched_file.buffer.flush
+          watched_file.listener.accept(line) unless line.empty?
+          watched_file.listener.eof
+          watched_file.file_close
+          key = watched_file.sincedb_key
+          sincedb_collection.reading_completed(key)
+          sincedb_collection.clear_watched_file(key)
+          watched_file.listener.deleted
+          watched_file.unwatch
+        end
+      end
+    end
+
+    def controlled_read(watched_file, loop_control)
+      logger.trace("reading...", "iterations" => loop_control.count, "amount" => loop_control.size, "filename" => watched_file.filename)
+      loop_control.count.times do
+        begin
+          result = watched_file.read_extract_lines(loop_control.size) # expect BufferExtractResult
+          logger.info(result.warning, result.additional) unless result.warning.empty?
+          result.lines.each do |line|
+            watched_file.listener.accept(line)
+            # sincedb position is independent from the watched_file bytes_read
+            delta = line.bytesize + @settings.delimiter_byte_size
+            sincedb_collection.increment(watched_file.sincedb_key, delta)
           end
+        rescue EOFError
+          logger.error("controlled_read: eof error reading file", "path" => watched_file.path, "error" => e.inspect, "backtrace" => e.backtrace.take(8))
+          break
+        rescue Errno::EWOULDBLOCK, Errno::EINTR
+          logger.error("controlled_read: block or interrupt error reading file", "path" => watched_file.path, "error" => e.inspect, "backtrace" => e.backtrace.take(8))
+          watched_file.listener.error
+          break
+        rescue => e
+          logger.error("controlled_read: general error reading file", "path" => watched_file.path, "error" => e.inspect, "backtrace" => e.backtrace.take(8))
+          watched_file.listener.error
+          break
         end
       end
     end

data/lib/filewatch/read_mode/handlers/read_zip_file.rb

@@ -13,6 +13,11 @@ module FileWatch module ReadMode module Handlers
       add_or_update_sincedb_collection(watched_file) unless sincedb_collection.member?(watched_file.sincedb_key)
       # can't really stripe read a zip file, its all or nothing.
       watched_file.listener.opened
+      # what do we do about quit when we have just begun reading the zipped file (e.g. pipeline reloading)
+      # should we track lines read in the sincedb and
+      # fast forward through the lines until we reach unseen content?
+      # meaning that we can quit in the middle of a zip file
+      key = watched_file.sincedb_key
       begin
         file_stream = FileInputStream.new(watched_file.path)
         gzip_stream = GZIPInputStream.new(file_stream)
@@ -31,8 +36,8 @@ module FileWatch module ReadMode module Handlers
         logger.error("Cannot decompress the gzip file at path: #{watched_file.path}")
         watched_file.listener.error
       else
-        watched_file.update_bytes_read(watched_file.last_stat_size)
-        sincedb_collection.unset_watched_file(watched_file)
+        sincedb_collection.store_last_read(key, watched_file.last_stat_size)
+        sincedb_collection.request_disk_flush
         watched_file.listener.deleted
         watched_file.unwatch
       ensure
@@ -42,7 +47,7 @@ module FileWatch module ReadMode module Handlers
         close_and_ignore_ioexception(gzip_stream) unless gzip_stream.nil?
         close_and_ignore_ioexception(file_stream) unless file_stream.nil?
       end
-      sincedb_collection.unset_watched_file(watched_file)
+      sincedb_collection.clear_watched_file(key)
     end
 
     private