listen 2.7.11 → 2.7.12

data/Rakefile

@@ -2,4 +2,11 @@ require 'bundler/gem_tasks'
 require 'rspec/core/rake_task'
 
 RSpec::Core::RakeTask.new(:spec)
-task default: :spec
+
+if ENV["CI"] != "true"
+  require "rubocop/rake_task"
+  RuboCop::RakeTask.new(:rubocop)
+  task default: [:spec, :rubocop]
+else
+  task default: [:spec]
+end

data/TROUBLESHOOTING.md

@@ -0,0 +1,139 @@
+# Issues and troubleshooting
+
+## 3 steps before you start diagnosing problems
+
+These 3 steps will:
+* help quickly troubleshoot issues caused by obscure problems
+* help quickly identify the area of the problem (a full list is [below](#known-issues))
+* help you get familiar with listen's diagnostic mode
+* help you create relevant output before you submit an issue
+
+1) For effective troubleshooting set the `LISTEN_GEM_DEBUGGING=1` variable
+before starting listen.
+
+2) Verify polling works (see `force_polling` option).
+
+After starting listen, you should see something like:
+```
+INFO -- : Celluloid loglevel set to: 1
+INFO -- : Record.build(): 0.06773114204406738 seconds
+```
+
+(Listen uses [Celluloid](https://github.com/celluloid/celluloid) for logging, so if you don't see anything, `Celluloid.logger` might have been disabled by a different gem, e.g. sidekiq)
+
+If you don't see the line `Record.build()`:
+* and there's a lot of disk activity, you may have to wait a few seconds
+* you may be using an outdated version of Listen
+* listen may have got stuck on a recursive symlink, see #259
+
+3) Make changes e.g. `touch foo` or `echo "a" >> foo` (for troubleshooting, avoid using an editor which could generate too many misleading events)
+
+You should see something like:
+
+```
+INFO -- : listen: raw changes: [[:added, "/home/me/foo"]]
+INFO -- : listen: final changes: {:modified=>[], :added=>["/home/me/foo"], :removed=>[]}
+```
+
+"raw changes" contains changes collected during the `:wait_for_delay` and `:latency` intervals, while "final changes" is what listen decided are relevant changes (for better editor support).
+
+## Adapter-specific diagnostics
+
+Use the `LISTEN_GEM_DEBUGGING` set to `2` for additional info.
+
+E.g. you'll get:
+
+```
+INFO -- : Celluloid loglevel set to: 0
+DEBUG -- : Broadcaster: starting tcp server: 127.0.0.1:4000
+DEBUG -- : Adapter: considering TCP ...
+DEBUG -- : Adapter: considering polling ...
+DEBUG -- : Adapter: considering optimized backend...
+INFO -- : Record.build(): 0.0007264614105224609 seconds
+DEBUG -- : inotify: foo ([:create])
+DEBUG -- : raw queue: [:file, #<Pathname:/tmp/x>, "foo", {:change=>:added}]
+DEBUG -- : added: file:/tmp/x/foo ({:change=>:added})
+DEBUG -- : inotify: foo ([:attrib])
+DEBUG -- : raw queue: [:file, #<Pathname:/tmp/x>, "foo", {:change=>:modified}]
+DEBUG -- : inotify: foo ([:close_write, :close])
+DEBUG -- : raw queue: [:file, #<Pathname:/tmp/x>, "foo", {:change=>:modified}]
+DEBUG -- : modified: file:/tmp/x/foo ({:change=>:modified})
+DEBUG -- : modified: file:/tmp/x/foo ({:change=>:modified})
+INFO -- : listen: raw changes: [[:added, "/tmp/x/foo"]]
+INFO -- : listen: final changes: {:modified=>[], :added=>["/tmp/x/foo"], :removed=>[]}
+DEBUG -- : Callback took 4.410743713378906e-05 seconds
+```
+
+This shows:
+* host port listened to (for forwarding events)
+* the actual adapter used (here, it's "optimized backend")
+* the event received (here it's `:create` from rb-inotify)
+* "raw queue" - events queued for processing (collected during `:latency`)
+* "Callback took" - how long it took your app to process changes
+
+#### Known issues
+
+Here are common issues grouped by area in which they occur:
+
+1. System/OS
+  * [Update your Dropbox client](http://www.dropbox.com/downloading), if you have Dropbox installed.
+  * old MacOS (< 10.6)
+  * certain old versions of Ruby (try a newer Ruby on Windows for `wdm` and TCP mode to work)
+  * system limits
+    * threads for Celluloid (e.g. Virtual Machine CPU/RAM limitations)
+    * [inotify limits (Linux)](https://github.com/guard/listen/wiki/Increasing-the-amount-of-inotify-watchers)
+  * system in an inconsistent state or stuck (try rebooting/updating on Windows/Mac - seriously!)
+  * FSEvent bug: (http://feedback.livereload.com/knowledgebase/articles/86239)
+
+2. Installation/gems/config
+  * not running listen or your app (e.g. guard) with `bundle exec` first
+  * old version of listen
+  * problems with adapter gems (`wdm`, `rb-fsevent`, `rb-inotify`) not installed, not detected properly (Windows) or not in Gemfile (Windows)
+  * Celluloid actors are silently crashing (when no LISTEN_GEM_DEBUGGING variable is present)
+  * see the [Performance](https://github.com/guard/listen/blob/master/README.md#Performance) section in the README
+
+3. Filesystem
+  * VM shared folders and network folders (NFS, Samba, SSHFS) don't work with optimized backends (workaround: polling, [TCP mode](https://github.com/guard/listen/blob/master/README.md#forwarding-file-events-over-tcp), Vagrant's rsync-auto mode, rsync/unison)
+  * FAT/HFS timestamps have 1-second precision, which can cause polling and rb-fsevent to be very slow on large files (try `LISTEN_GEM_DISABLE_HASHING` variable)
+  * virtual filesystems may not implement event monitoring
+  * restrictive file/folder permissions
+  * watched folders moved/removed while listen was running (try restarting listen and moving/copying watched folder to a new location)
+
+4. Insufficient latency (for polling and rb-fsevent)
+  * too many files being watched (polling) and not enough threads or CPU power
+  * slow editor save (see below)
+  * slow hard drive
+  * encryption
+  * a combination of factors
+
+5. Too few or too many callbacks (`:wait_for_delay` option)
+  * complex editor file-save causes events to happen during callback (result: multiple callbacks if wait_for_delay is too small)
+  * too large when using TCP mode (see timestamps in output to debug)
+  * too many changes happening too frequently (use ignore rules to filter them out)
+
+6. Paths
+  * default ignore rules
+  * encoding-related issues (bad filenames, mounted FS encoding mismatch)
+  * symlinks may cause listen to hang (#259)
+  * symlinks may not work as you expect or work differently for polling vs non-polling
+  * TCP paths don't match with client's current working directory
+
+7. Editors
+  * "atomic save" in editors may confuse listen (disable it and try again)
+  * listen's default ignore rules may need tweaking
+  * your editor may not be supported yet (see default ignore rules for editors)
+  * use `touch foo` or `echo "a" >> foo`  to confirm it's an editor issue
+  * slow terminal/GFX card, slow font, transparency effects in terminal
+  * complex/lengthy editor save (due to e.g. many plugins running during save)
+  * listen has complex rules for detecting atomic file saves (Linux)
+
+8. TCP (tcp mode) issues
+  * not a recent listen gem (before 2.7.11)
+  * additional network delay and collecting may need a higher `:wait_for_delay` value
+  * changes (added, removed, deleted) not matching actual changes
+
+If your application keeps using the polling-adapter and you can't figure out why, feel free to [open an issue](https://github.com/guard/listen/issues/new) (and be sure to [give all the details](https://github.com/guard/listen/blob/master/CONTRIBUTING.md)).
+
+Listen traps SIGINT signal to properly finalize listeners. If you plan
+on trapping this signal yourself - make sure to call `Listen.stop` in
+signal handler.

data/lib/listen.rb

@@ -1,6 +1,8 @@
 require 'celluloid'
 require 'listen/listener'
 
+require 'listen/internals/thread_pool'
+
 module Listen
   class << self
     # Listens to file system modifications on a either single directory or
@@ -21,11 +23,9 @@ module Listen
     def to(*args, &block)
       Celluloid.boot unless Celluloid.running?
       options = args.last.is_a?(Hash) ? args.last : {}
-      if target = options.delete(:forward_to)
-        _add_listener(target, :broadcaster, *args, &block)
-      else
-        _add_listener(*args, &block)
-      end
+      target = options.delete(:forward_to)
+      args = ([target, :broadcaster] + args) if target
+      _add_listener(*args, &block)
     end
 
     # Stop all listeners & Celluloid
@@ -36,6 +36,7 @@ module Listen
     # This is used by the `listen` binary to handle Ctrl-C
     #
     def stop
+      Internals::ThreadPool.stop
       @listeners ||= []
 
       # TODO: should use a mutex for this

data/lib/listen/adapter.rb

@@ -22,7 +22,8 @@ module Listen
       _warn_polling_fallback(options)
       Polling
     rescue
-      _log :warn, "Adapter: failed: #{$!.inspect}:#{$@.join("\n")}"
+      _log :warn, format('Adapter: failed: %s:%s', $ERROR_POSITION.inspect,
+                         $ERROR_POSITION * "\n")
       raise
     end
 

data/lib/listen/adapter/base.rb

@@ -28,7 +28,7 @@ module Listen
         defaults = self.class.const_get('DEFAULTS')
         @options = Listen::Options.new(options, defaults)
       rescue
-        _log :error, "adapter config failed: #{$!}:#{$@.join("\n")}"
+        _log_exception 'adapter config failed: %s:%s'
         raise
       end
 
@@ -53,11 +53,11 @@ module Listen
 
       def start
         configure
-        Thread.new do
+        Listen::Internals::ThreadPool.add do
           begin
             _run
           rescue
-            _log :error, "run() in thread failed: #{$!}:#{$@.join("\n")}"
+            _log_exception 'run() in thread failed: %s:%s'
             raise
           end
         end
@@ -83,6 +83,10 @@ module Listen
         self.class.send(:_log, *args)
       end
 
+      def _log_exception(msg)
+        _log :error, format(msg, $ERROR_INFO, $ERROR_POSITION * "\n")
+      end
+
       def self._log(*args)
         Celluloid::Logger.send(*args)
       end

data/lib/listen/adapter/darwin.rb

@@ -1,3 +1,5 @@
+require 'listen/internals/thread_pool'
+
 module Listen
   module Adapter
     # Adapter implementation for Mac OS X `FSEvents`.
@@ -10,6 +12,7 @@ module Listen
 
       private
 
+      # NOTE: each directory gets a DIFFERENT callback!
       def _configure(dir, &callback)
         require 'rb-fsevent'
         opts = { latency: options.latency }
@@ -20,8 +23,20 @@ module Listen
         end
       end
 
+      # NOTE: _run is called within a thread, so run every other
+      # worker in it's own thread
       def _run
-        @workers.pop.run while !@workers.empty?
+        first = @workers.pop
+        until @workers.empty?
+          Listen::Internals::ThreadPool.add do
+            begin
+              @workers.pop.run
+            rescue
+              _log_exception 'run() in extra thread(s) failed: %s: %s'
+            end
+          end
+        end
+        first.run
       end
 
       def _process_event(dir, event)

data/lib/listen/adapter/tcp.rb

@@ -26,16 +26,18 @@ module Listen
         @buffer = ''
         async.run
       rescue Celluloid::Task::TerminatedError
-        _log :debug, "TCP adapter was terminated: #{$!.inspect}"
+        _log :debug, "TCP adapter was terminated: #{$ERROR_INFO.inspect}"
       rescue Errno::ECONNREFUSED
         sleep 1
         attempts -= 1
-        _log :warn, "TCP.start: #{$!.inspect}"
+        _log :warn, "TCP.start: #{$ERROR_INFO.inspect}"
         retry if attempts > 0
-        _log :error, "TCP.start: #{$!.inspect}:#{$@.join("\n")}"
+        _log :error, format('TCP.start: %s:%s', $ERROR_INFO.inspect,
+                            $ERROR_POSITION * "\n")
         raise
       rescue
-        _log :error, "TCP.start: #{$!.inspect}:#{$@.join("\n")}"
+        _log :error, format('TCP.start: %s:%s', $ERROR_INFO.inspect,
+                            $ERROR_POSITION * "\n")
         raise
       end
 
@@ -65,7 +67,8 @@ module Listen
           handle_message(message)
         end
       rescue
-        _log :error, "TCP.handle_data crashed: #{$!}:#{$@.join("\n")}"
+        _log :error, format('TCP.handle_data crashed: %s:%s', $ERROR_INFO,
+                            $ERROR_POSITION * "\n")
         raise
       end
 

data/lib/listen/adapter/windows.rb

@@ -7,10 +7,7 @@ module Listen
 
       BUNDLER_DECLARE_GEM = <<-EOS.gsub(/^ {6}/, '')
         Please add the following to your Gemfile to avoid polling for changes:
-          require 'rbconfig'
-          if RbConfig::CONFIG['target_os'] =~ /mswin|mingw|cygwin/i
-            gem 'wdm', '>= 0.1.0'
-          end
+          gem 'wdm', '>= 0.1.0' if Gem.win_platform?
       EOS
 
       def self.usable?
@@ -18,7 +15,9 @@ module Listen
         require 'wdm'
         true
       rescue LoadError
-        _log :debug, "wdm - load failed: #{$!}:#{$@.join("\n")}"
+        _log :debug, format('wdm - load failed: %s:%s', $ERROR_INFO,
+                            $ERROR_POSITION * "\n")
+
         Kernel.warn BUNDLER_DECLARE_GEM
         false
       end
@@ -82,7 +81,8 @@ module Listen
         end
       rescue
         details = event.inspect
-        _log :error, "wdm - callback (#{details}): #{$!}:#{$@.join("\n")}"
+        _log :error, format('wdm - callback (%): %s:%s', details, $ERROR_INFO,
+                            $ERROR_POSITION * "\n")
         raise
       end
 

data/lib/listen/change.rb

@@ -43,9 +43,10 @@ module Listen
         end
       end
     rescue Celluloid::Task::TerminatedError
-      _log :debug, "Change#change was terminated: #{$!.inspect}"
+      _log :debug, "Change#change was terminated: #{$ERROR_INFO.inspect}"
     rescue RuntimeError
-      _log :error, "Change#change crashed #{$!.inspect}:#{$@.join("\n")}"
+      _log :error, format('Change#change crashed %s:%s', $ERROR_INFO.inspect,
+                          $ERROR_POSITION * "\n")
       raise
     end
 

data/lib/listen/cli.rb

@@ -55,9 +55,7 @@ module Listen
       listener = Listen.to directory, forward_to: address, &callback
       listener.start
 
-      while listener.listen?
-        sleep 0.5
-      end
+      sleep 0.5 while listener.listen?
     end
   end
 end

data/lib/listen/directory.rb

@@ -13,16 +13,10 @@ module Listen
       path = dir + rel_path
       current = Set.new(path.children)
 
-      if options[:silence]
-        _log(:debug) do
-          "Recording: #{rel_path}: #{options.inspect}"\
-            " [#{previous.inspect}] -> (#{current.inspect})"
-        end
-      else
-        _log(:debug) do
-          "Scanning: #{rel_path}: #{options.inspect}"\
-          " [#{previous.inspect}] -> (#{current.inspect})"
-        end
+      _log(:debug) do
+        format('%s: %s(%s): %s -> %s',
+               (options[:silence] ? 'Recording' : 'Scanning'),
+               rel_path, options.inspect, previous.inspect, current.inspect)
       end
 
       current.each do |full_path|
@@ -46,7 +40,9 @@ module Listen
       _async_changes(dir, path, queue, previous, options)
       _change(queue, :file, dir, rel_path, options)
     rescue
-      _log(:warn) { "scanning DIED: #{$!}:#{$@.join("\n")}" }
+      _log(:warn) do
+        format('scan DIED: %s:%s', $ERROR_INFO, $ERROR_POSITION * "\n")
+      end
       raise
     end
 

data/lib/listen/file.rb

@@ -23,52 +23,46 @@ module Listen
         return :modified
       end
 
-      unless /1|true/ =~ ENV['LISTEN_GEM_DISABLE_HASHING']
-        if self.inaccurate_mac_time?(lstat)
-          # Check if change happened within 1 second (maybe it's even
-          # too much, e.g. 0.3-0.5 could be sufficient).
-          #
-          # With rb-fsevent, there's a (configurable) latency between
-          # when file was changed and when the event was triggered.
-          #
-          # If a file is saved at ???14.998, by the time the event is
-          # actually received by Listen, the time could already be e.g.
-          # ???15.7.
-          #
-          # And since Darwin adapter uses directory scanning, the file
-          # mtime may be the same (e.g. file was changed at ???14.001,
-          # then at ???14.998, but the fstat time would be ???14.0 in
-          # both cases).
-          #
-          # If change happend at ???14.999997, the mtime is 14.0, so for
-          # an mtime=???14.0 we assume it could even be almost ???15.0
-          #
-          # So if Time.now.to_f is ???15.999998 and stat reports mtime
-          # at ???14.0, then event was due to that file'd change when:
-          #
-          # ???15.999997 - ???14.999998 < 1.0s
-          #
-          # So the "2" is "1 + 1" (1s to cover rb-fsevent latency +
-          # 1s maximum difference between real mtime and that recorded
-          # in the file system)
-          #
-          if data[:mtime].to_i + 2 > Time.now.to_f
-            begin
-              md5 = Digest::MD5.file(path).digest
-              record.async.update_file(dir, rel_path, data.merge(md5: md5))
-              :modified if record_data[:md5] && md5 != record_data[:md5]
+      return if /1|true/ =~ ENV['LISTEN_GEM_DISABLE_HASHING']
+      return unless self.inaccurate_mac_time?(lstat)
 
-            rescue SystemCallError
-              # ignore failed md5
-            end
-          end
-        end
-      end
+      # Check if change happened within 1 second (maybe it's even
+      # too much, e.g. 0.3-0.5 could be sufficient).
+      #
+      # With rb-fsevent, there's a (configurable) latency between
+      # when file was changed and when the event was triggered.
+      #
+      # If a file is saved at ???14.998, by the time the event is
+      # actually received by Listen, the time could already be e.g.
+      # ???15.7.
+      #
+      # And since Darwin adapter uses directory scanning, the file
+      # mtime may be the same (e.g. file was changed at ???14.001,
+      # then at ???14.998, but the fstat time would be ???14.0 in
+      # both cases).
+      #
+      # If change happend at ???14.999997, the mtime is 14.0, so for
+      # an mtime=???14.0 we assume it could even be almost ???15.0
+      #
+      # So if Time.now.to_f is ???15.999998 and stat reports mtime
+      # at ???14.0, then event was due to that file'd change when:
+      #
+      # ???15.999997 - ???14.999998 < 1.0s
+      #
+      # So the "2" is "1 + 1" (1s to cover rb-fsevent latency +
+      # 1s maximum difference between real mtime and that recorded
+      # in the file system)
+      #
+      return if data[:mtime].to_i + 2 <= Time.now.to_f
+
+      md5 = Digest::MD5.file(path).digest
+      record.async.update_file(dir, rel_path, data.merge(md5: md5))
+      :modified if record_data[:md5] && md5 != record_data[:md5]
     rescue SystemCallError
       record.async.unset_path(dir, rel_path)
       :removed
     rescue
-      Celluloid::Logger.debug "lstat failed for: #{rel_path} (#{$!})"
+      Celluloid::Logger.debug "lstat failed for: #{rel_path} (#{$ERROR_INFO})"
       raise
     end