em-dir-watcher 0.1.0 → 0.9.1

data/LICENSE

@@ -1,4 +1,5 @@
-Copyright (c) 2009 Andrey Tarantsov
+Copyright (c) 2010 Andrey Tarantsov
+Copyright (c) 2010 Mikhail Gusarov
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.md

@@ -1,7 +1,46 @@
-em-dir-watcher: real directory monitoring for EventMachine
-==========================================================
+em-dir-watcher: sane cross-platform file system change monitoring for Event Machine
+===================================================================================
+
+Compatible with Ruby 1.8+
+
+Supported platforms:
+
+* Mac OS X: employs FSEvents API via Ruby Cocoa. To support Ruby 1.8, forks a separate watcher process and communicates with it via pipes. (Ruby 1.9 should use threads instead, but this hasn't been implemented yet.)
+
+* Linux: employs inotify via rb-inotify gem. This backend is nicest of all because inotify uses file descriptors and thus can be monitored from within Event Machine reactor.
+
+* Windows: employs Directory Change Notifications API via win32-changenotify. To support Ruby 1.8, forks a separate watcher process and communicates with it via sockets. (We'd love to communicate via pipes, but couldn't figure a way to do it in a Event Machine-friendly way. Also we'd love to use threads on Ruby 1.9, but not there yet.)
+
+* There is no fallback polling backend at the moment. Please contribute it if you need one.
+
+
+Why not FSSM or Directory Watcher?
+--------------------------------------
+
+FSSM rescans the entire directory tree on each change, so it has about 0.5 sec lag on average-sized projects. Em-dir-watcher only rescans the changed subdirectories (on all systems), and avoids rescanning subtrees on the systems that support non-subtree notifications (Mac, Linux). We'd love to see our `Tree` class used in fssm — this should be an easy change.
+
+Also, fssm does not know anything about Event Machine, so has to be run in a separate process/thread even on the systems that are reactor-friendly (i.e. Linux). Em-dir-watcher uses `EM.watch` to listen to inotify events on Linux.
+
+DirectoryWatcher's Event Machine edition uses `EM.watchFile`, which runs out of max open file limit pretty quickly. Also it employs polling to catch added or removed files, and has to walk the entire directory tree every time. Em-dir-watcher uses native backends to watch for file system changes (just like FSSM), and finds added or removed files just as quickly as modified ones.
+
+Besides, both fssm and directory_watcher do not support exclusions, and thus will walk, update and keep the entire tree in memory including the subfolders you don't need. Em-dir-watcher never walks excluded subfolders, so you can exclude the stuff you don't need to watch to further improve the performance.
+
+
+Installation
+------------
+
+Mac:
+
+    sudo gem install em-dir-watcher
+
+Linux:
+
+    sudo gem install rb-inotify em-dir-watcher
+
+Windows:
+
+    gem install win32-changenotify em-dir-watcher
 
-Employs FSEvents, inotify or Win32 Directory Change Notifications APIs under EventMachine. (Forks a subprocess for blocking watchers.)
 
 Usage
 -----
@@ -10,11 +49,13 @@ Usage
     require 'em-dir-watcher'
 
     EM.run {
-        dw = EMDirWatcher.watch '.', ['**/*.css', 'lib/**/*.rb'] do |path|
-            if File.exists? path
-                puts "Modified: #{path}"
-            else
-                puts "Deleted: #{path}"
+        dw = EMDirWatcher.watch '.' do |paths|
+            paths.each do |path|
+                if File.exists? path
+                    puts "Modified: #{path}"
+                else
+                    puts "Deleted: #{path}"
+                end
             end
         end
         puts "EventMachine running..."
@@ -22,7 +63,83 @@ Usage
 
 Run `examples/monitor.rb` to see it in action.
 
+
+EMDirWatcher.watch
+------------------
+
+`EMDirWatcher.watch` accepts a path and an options hash:
+
+    EMDirWatcher.watch File.expand_path('~/my_project'),
+        :include_only => ['*.html', '*.css', '*.js'],
+        :exclude => ['~*', 'vendor/plugins'],
+        :grace_period => 0.2
+
+It returns an object that has a single `stop` method:
+
+    dw = EMDirWatcher.watch File.expand_path('~/my_project')
+    ...
+    dw.stop
+
+
+Inclusions and exclusions
+-------------------------
+
+If the list of inclusions is `nil` (the default), all files are included. Otherwise it has to be an array, and specifies a list of patterns to monitor. Each pattern is either a name glob, a path glob or a path regexp (more on this later).
+
+The list of exclusions defaults to an empty array, and specifies the list of patterns to exclude. Each pattern is either a name glob, a path glob or a path regexp. If a path matches both an inclusion and an exclusion filter, it is excluded.
+
+Patterns are inspired by Git's `.gitignore` conventions. Each pattern can be one of the following:
+
+* A string that does not contain a slash is treated as a shell-style glob and is matched against file **base names.** For example, you can use `*.html` to match HTML files in any directory, or `~*` to match temporary files in any directory.
+
+* A string that contains a slash is treated as a shell-style glob and is matched against file **paths** (relative to the monitored directory). For example, you can use `lib/rake` to match all files in `lib/rake` directory, or `/vendor` to match a `vendor` directory.
+
+* A regexp is matched against file relative paths. For example, you can use `/[A-Z]/` to match all files and directories that contain upper-case characters in their name.
+
+
+Grace period
+------------
+
+Use `:grace_period => 1.0` to combine the changes together if they occur in quick succession. This, for example, may be used to avoid starting a build while some files are still being updated from a repository.
+
+The default grace period is `0`, which means that changes are reported immediately when they occur.
+
+
+Hacking
+-------
+
+Required software:
+
+    sudo gem install jeweler shoulda
+
+To run the tests, use:
+
+    rake test
+
+This is expected to work on all platforms and shouldn't give any failures, EXCEPT that on a Mac spurious test failures occur in about 1–3 of 20 test runs, and we are unable to get rid of them.
+
+You can use `./testloop` script to run the tests multiple times in a row to check for unreliable behaviors.
+
+To give a more context on Mac test failures, two constants that have an effect on them are `STARTUP_DELAY` in `lib/em-dir-watcher/platform/mac.rb` and `UNIT_DELAY` in `tests/test_monitor.rb`. We've settled on a sweet spot of `0.5`/`0.5`, which gives a 5%–15% failure rate. Increasing them to `1.0`/`1.0` still results in the same failure rate. Decreasing them to `0.2`/`0.2` results in 30% failed test runs.
+
+You can use `rake rcov` to check code coverage info. Currently `tree.rb` and `monitor.rb` have 100% test coverage, `mac.rb` has 80% coverage, `linux.rb` should have about 100% coverage and `windows.rb` should have about 80% coverage (the last two were not measured).
+
+
+Help Wanted aka TODO
+--------------------
+
+If you find yourself using this gem, consider implementing one of the following functions:
+
+* Ruby 1.9-friendly thread-based Mac backend
+* Ruby 1.9-friendly thread-based Windows backend
+* FFI-based Mac backend — there is no need to employ Ruby Cocoa monster just to invoke a handful of functions; a nearly-working version is in `lib/em-dir-watcher/platform/mac/ffi_fsevents_watcher.rb`, just needs some final touches
+* polling-based fallback backend for the folks on funny operating systems (from FreeBSD to Solaris to HP UX :)
+* use Mac OS X's FSEvents native grace period implementation
+
+
 License
 -------
 
-Copyright (c) 2010 Andrey Tarantsov. Distributed under the MIT license. See LICENSE for details.
+Copyright © 2010 Andrey Tarantsov, Mikhail Gusarov.
+
+Distributed under the MIT license. See LICENSE for details.

data/Rakefile

@@ -9,7 +9,7 @@ begin
     gem.description = gem.summary
     gem.email = "andreyvit@gmail.com"
     gem.homepage = "http://github.com/mockko/em-dir-watcher"
-    gem.authors = ["Andrey Tarantsov"]
+    gem.authors = ["Andrey Tarantsov", "Mikhail Gusarov"]
 #     gem.add_development_dependency "thoughtbot-shoulda", ">= 0"
     # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
   end

data/VERSION

@@ -1 +1 @@
-0.1.0
+0.9.1

data/examples/monitor.rb

@@ -4,11 +4,19 @@ require 'rubygems'
 require 'em-dir-watcher'
 
 dir = (ARGV.empty? ? '.' : ARGV.shift)
-globs = (ARGV.empty? ? ['**/*'] : ARGV)
+inclusions = ARGV.reject { |arg| arg =~ /^!/ }
+inclusions = nil if inclusions == []
+exclusions = ARGV.select { |arg| arg =~ /^!/ }.collect { |arg| arg[1..-1] }
+
+EM.error_handler{ |e|
+    puts "Error raised during event loop: #{e.class.name} #{e.message}"
+    puts e.backtrace
+}
 
 EM.run {
-    dw = EMDirWatcher.watch dir, globs do |path|
-        if File.exists? path
+    dw = EMDirWatcher.watch dir, inclusions, exclusions do |path|
+        full_path = File.join(dir, path)
+        if File.exists? full_path
             puts "Modified: #{path}"
         else
             puts "Deleted: #{path}"

data/lib/em-dir-watcher/invokers/subprocess_invoker.rb

@@ -0,0 +1,100 @@
+
+require 'io/nonblock'
+
+module EMDirWatcher
+module Invokers
+
+class SubprocessInvoker
+
+    attr_reader :active
+    attr_reader :input_handler
+    attr_accessor :additional_delay
+
+    def initialize subprocess, &input_handler
+        @subprocess = subprocess
+        @input_handler = input_handler
+        @active = true
+        @ready_to_use = false
+        @ready_to_use_handlers = []
+        @additional_delay = 0.1
+        start_subprocess
+    end
+
+    def when_ready_to_use &ready_to_use_handler
+      if @ready_to_use_handlers.nil?
+          ready_to_use_handler.call()
+      else
+          @ready_to_use_handlers << ready_to_use_handler
+      end
+    end
+
+    def stop
+        @active = false
+        kill
+    end
+
+    # private methods
+
+    def ready_to_use!
+        return if @ready_to_use
+        @ready_to_use = true
+        EM.add_timer @additional_delay do
+            @ready_to_use_handlers.each { |handler| handler.call() }
+            @ready_to_use_handlers = nil
+        end
+    end
+
+    def kill
+        if @io
+            Process.kill 'TERM', @io.pid 
+            Process.waitpid @io.pid
+            @io = nil
+        end
+    end
+
+    def start_subprocess
+        io = open('|-', 'r')
+        if io.nil?
+            $stdout.sync = true
+            ready  = lambda { puts }
+            output = lambda { |single_line_string| puts single_line_string.strip }
+            @subprocess.call ready, output
+            exit
+        end
+        @io = io
+        @io.nonblock = true
+
+        @connection = EM.watch io do |conn|
+            class << conn
+                attr_accessor :invoker
+                
+                def notify_readable
+                    @invoker.ready_to_use!
+                    @data_received ||= ""
+                    @data_received << @io.read
+                    while line = @data_received.slice!(/^[^\n]*[\n]/m)
+                        @invoker.input_handler.call line.strip
+                    end
+                rescue EOFError
+                    detach
+                    @invoker.kill  # waitpid to cleanup zombie
+                    if @invoker.active
+                        EM.next_tick do
+                            @invoker.start_subprocess
+                        end
+                    end
+                end
+
+                def unbind
+                    @invoker.kill
+                end
+            end
+            conn.invoker = self
+            conn.notify_readable = true
+        end
+    end
+
+end
+
+end
+end

data/lib/em-dir-watcher/monitor.rb

@@ -0,0 +1,47 @@
+require 'set'
+
+module EMDirWatcher
+    Watcher = Platform.const_get(PLATFORM)::Watcher
+
+    DEFAULT_OPTIONS = {
+        :exclude => [],
+        :include_only => nil,
+        :grace_period => 0.0,
+    }.freeze
+
+    INFINITELY_SMALL_PERIOD = 0.001
+
+    def self.watch path, options={}
+        unless (invalid_keys = options.keys - DEFAULT_OPTIONS.keys).empty?
+            raise StandardError, "Unsupported options given to EMDirWatcher.watch: " + invalid_keys.join(", ")
+        end
+        options = DEFAULT_OPTIONS.merge(options)
+        grace_period = options[:grace_period]
+
+        tree = Tree.new path, options[:include_only], options[:exclude]
+
+        pending_refresh_requests = Set.new
+        process_pending_refresh_requests_scheduled = false
+
+        process_pending_refresh_requests = lambda do
+            process_pending_refresh_requests_scheduled = false
+            changed_paths = Set.new
+            pending_refresh_requests.each do |change_scope, refresh_subtree|
+                changed_paths += tree.refresh! change_scope, refresh_subtree
+            end
+            yield changed_paths.to_a unless changed_paths.empty?
+        end
+
+        Watcher.new path, options[:include_only], options[:exclude] do |change_scope, refresh_subtree|
+            pending_refresh_requests << [change_scope, refresh_subtree]
+            if grace_period <= INFINITELY_SMALL_PERIOD
+                process_pending_refresh_requests.call
+            else
+                unless process_pending_refresh_requests_scheduled
+                    EM.add_timer grace_period, &process_pending_refresh_requests
+                    process_pending_refresh_requests_scheduled = true
+                end
+            end
+        end
+    end
+end

data/lib/em-dir-watcher/platform/linux.rb

@@ -0,0 +1,51 @@
+require 'rb-inotify'
+require 'io/nonblock'
+
+module EMDirWatcher
+module Platform
+module Linux
+
+module Native
+  extend FFI::Library
+  ffi_lib "c"
+  attach_function :close, [:int], :int
+end
+
+class Watcher
+
+  def initialize path, inclusions, exclusions
+    @notifier = INotify::Notifier.new
+
+    @notifier.watch(path, :recursive, :attrib, :modify, :create,
+                    :delete, :delete_self, :moved_from, :moved_to,
+                    :move_self) do |event|
+      yield event.absolute_name
+    end
+
+    @conn = EM.watch @notifier.to_io do |conn|
+      class << conn
+        attr_accessor :notifier
+
+        def notify_readable
+          @notifier.process
+        end
+      end
+      conn.notifier = @notifier
+      conn.notify_readable = true
+    end
+  end
+
+  def when_ready_to_use
+      yield
+  end
+
+  def ready_to_use?; true; end
+
+  def stop
+    Native.close @notifier.fd
+  end
+end
+
+end
+end
+end

data/lib/em-dir-watcher/platform/mac/ffi_fsevents_watcher.rb

@@ -0,0 +1,80 @@
+
+require 'ffi'
+
+module EMDirWatcher
+module Platform
+module Mac
+
+module CarbonCore
+    extend FFI::Library
+    ffi_lib '/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Versions/Current/CarbonCore'
+
+    attach_function :CFStringCreateWithCString, [:pointer, :string, :int], :pointer
+    KCFStringEncodingUTF8 = 0x08000100
+    attach_function :CFStringGetLength, [:pointer, :pointer, :int, :pointer, :pointer, :pointer], :int
+
+    attach_function :CFArrayCreate, [:pointer, :pointer, :int, :pointer], :pointer
+
+    attach_function :CFRunLoopRun, :CFRunLoopRun, [], :void
+    attach_function :CFRunLoopGetCurrent, [], :pointer
+    attach_variable :kCFRunLoopDefaultMode, :pointer
+
+    callback :FSEventStreamCallback, [:int], :void
+
+    KFSEventStreamEventIdSinceNow = -1
+    attach_function :FSEventStreamCreate, [:pointer, :FSEventStreamCallback, :pointer, :pointer, :long, :double, :int], :pointer
+    attach_function :FSEventStreamScheduleWithRunLoop, [:pointer, :pointer, :pointer], :void
+    attach_function :FSEventStreamStart, [:pointer], :void
+    attach_function :FSEventStreamStop, [:pointer], :void
+end
+
+class FSEventStream
+    
+    KFSEventStreamEventFlagMustScanSubDirs = 0x1
+
+    class StreamError < StandardError;
+    end
+
+    def initialize(paths, &block)
+        raise ArgumentError, 'No callback block was specified.' unless block_given?
+        paths.each { |path| raise ArgumentError, "The specified path (#{path}) does not exist." unless File.exist?(path) }
+
+        handler = lambda do |stream, client_callback_info, number_of_events, paths_pointer, event_flags, event_ids|
+            $stderr.puts "CHANGED!"
+            block.call(['/'])
+        end
+        latency = 0.0
+        flags = 0
+
+        path_cfstring = CarbonCore.CFStringCreateWithCString nil, paths[0], CarbonCore::KCFStringEncodingUTF8
+        # puts "path_cfstring = #{path_cfstring}"
+        # puts "len = #{CarbonCore.CFStringGetLength(path_cfstring)}"
+
+        paths_ptr = FFI::MemoryPointer.new(:pointer)
+        paths_ptr.write_pointer path_cfstring
+        paths_cfarray = CarbonCore.CFArrayCreate nil, paths_ptr, 1, nil
+        # puts "paths_cfarray = #{paths_cfarray}"
+
+        fsevent_stream = CarbonCore.FSEventStreamCreate nil, handler, nil, paths_cfarray, CarbonCore::KFSEventStreamEventIdSinceNow, 0.0, 0
+        # puts "fsevent_stream = #{fsevent_stream}"
+
+        # puts "CarbonCore.kCFRunLoopDefaultMode = #{CarbonCore.kCFRunLoopDefaultMode}"
+        # puts "len = #{CarbonCore.CFStringGetLength(CarbonCore.kCFRunLoopDefaultMode)}"
+
+        CarbonCore.FSEventStreamScheduleWithRunLoop fsevent_stream, CarbonCore.CFRunLoopGetCurrent, CarbonCore.kCFRunLoopDefaultMode
+
+        CarbonCore.FSEventStreamStart fsevent_stream
+    end
+    
+    def run_loop
+        CarbonCore.CFRunLoopRun
+    end
+
+    def stop
+        CarbonCore.FSEventStreamStop(@stream)
+    end
+end
+
+end
+end
+end

data/lib/em-dir-watcher/platform/mac/rubycocoa_watcher.rb

@@ -0,0 +1,51 @@
+
+require 'osx/foundation'
+OSX.require_framework '/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework'
+
+module EMDirWatcher
+module Platform
+module Mac
+
+class FSEventStream
+    
+    KFSEventStreamEventFlagMustScanSubDirs = 0x1
+
+    class StreamError < StandardError;
+    end
+
+    def initialize(paths, &block)
+        raise ArgumentError, 'No callback block was specified.' unless block_given?
+        paths.each { |path| raise ArgumentError, "The specified path (#{path}) does not exist." unless File.exist?(path) }
+
+        callback = Proc.new do |stream, client_callback_info, number_of_events, paths_pointer, event_flags, event_ids|
+            paths_pointer.regard_as('*')
+            # event_flags.regard_as('*')
+            events = []
+            number_of_events.times {|i|
+                flags = event_flags[i]
+                code = if (flags & KFSEventStreamEventFlagMustScanSubDirs) == KFSEventStreamEventFlagMustScanSubDirs then '>' else '-' end
+                events << code + paths_pointer[i].to_s
+            }
+            block.call(events)
+        end
+        latency = 0.0
+        flags = 0
+        @stream = OSX.FSEventStreamCreate(OSX::KCFAllocatorDefault, callback, nil, paths, OSX::KFSEventStreamEventIdSinceNow, latency, flags)
+        raise(StreamError, 'Unable to create FSEvents stream.') unless @stream
+        OSX.FSEventStreamScheduleWithRunLoop(@stream, OSX.CFRunLoopGetCurrent, OSX::KCFRunLoopDefaultMode)
+        ok = OSX.FSEventStreamStart(@stream)
+        raise(StreamError, 'Unable to start FSEvents stream.') unless ok
+    end
+    
+    def run_loop
+        OSX.CFRunLoopRun
+    end
+
+    def stop
+        OSX.FSEventStreamStop(@stream)
+    end
+end
+
+end
+end
+end

data/lib/em-dir-watcher/platform/mac.rb

@@ -0,0 +1,50 @@
+
+require "em-dir-watcher/invokers/subprocess_invoker"
+
+module EMDirWatcher
+module Platform
+module Mac
+
+class Watcher
+
+    STARTUP_DELAY = 0.5
+
+    attr_accessor :handler, :active
+
+    def initialize path, inclusions, exclusions
+        subprocess = lambda do |ready, output|
+            require "em-dir-watcher/platform/mac/rubycocoa_watcher"
+            # require "em-dir-watcher/platform/mac/ffi_fsevents_watcher"
+            stream = FSEventStream.new [path] do |changed_paths|
+                changed_paths.each { |path| output.call path }
+            end
+            ready.call()
+            stream.run_loop
+        end
+        
+        @invoker = EMDirWatcher::Invokers::SubprocessInvoker.new subprocess do |path|
+            code, path = path[0], path[1..-1]
+            if code == ?> || code == ?-
+                refresh_subtree = (code == ?>)
+                yield path, refresh_subtree
+            end
+        end
+        # Mac OS X seems to require this delay till it really starts listening for file system changes.
+        # See README for explaination of the effect.
+        @invoker.additional_delay = STARTUP_DELAY
+    end
+
+    def when_ready_to_use &ready_to_use_handler
+        @invoker.when_ready_to_use &ready_to_use_handler
+    end
+
+    def ready_to_use?; true; end
+
+    def stop
+        @invoker.stop
+    end
+
+end
+end
+end
+end

data/lib/em-dir-watcher/platform/windows/monitor.rb

@@ -1,5 +1,4 @@
 $stdout.sync = true
-
 require 'rubygems'
 require 'socket'
 require 'win32/changenotify'
@@ -14,7 +13,6 @@ socket = TCPSocket.open('127.0.0.1', port)
 begin
   cn.wait do |events|
     events.each do |event|
-      # puts event.file_name
       socket.puts event.file_name
     end
   end

data/lib/em-dir-watcher/platform/windows/path_to_ruby_exe.rb

@@ -13,10 +13,7 @@ public
 
   def self.path_to_ruby_exe
     buf = 0.chr * 260
-    len = [buf.length].pack('L')
-
     GetModuleFileName.call(0, buf, buf.length)
-
     buf.strip 
   end
 

data/lib/em-dir-watcher/platform/windows.rb

@@ -1,5 +1,6 @@
 
 require File.join(File.dirname(__FILE__), 'windows', 'path_to_ruby_exe')
+require 'socket'
 
 module EMDirWatcher
 module Platform
@@ -21,23 +22,44 @@ module TcpHandler
 end
 
 class Watcher
-
-    def initialize path, globs, &handler
+    
+    def initialize path, inclusions, exclusions, &handler
         @path = path
-        @globs = globs
         @handler = handler
         @active = true
+        @ready_to_use = false
+        @ready_to_use_handlers = []
 
         start_server
         setup_listener
     end
 
+    def when_ready_to_use &ready_to_use_handler
+      if @ready_to_use_handlers.nil?
+          ready_to_use_handler.call()
+      else
+          @ready_to_use_handlers << ready_to_use_handler
+      end
+    end
+
+    def ready_to_use!
+        return if @ready_to_use
+        @ready_to_use = true
+        # give the child process additional 100ms to start watching loop
+        EM.add_timer 0.1 do
+            @ready_to_use_handlers.each { |handler| handler.call() }
+            @ready_to_use_handlers = nil
+        end
+    end
+    
+    def ready_to_use?; @ready_to_use; end
+
     def start_server
         @server = EM.start_server '127.0.0.1', 0, TcpHandler do |server|
             server.watcher = self
+            ready_to_use!
         end
-        @server_port, _ = Socket.unpack_sockaddr_in(EM::get_sockname @server)
-        puts "Server running on port #{@server_port}"
+        @server_port, _ = Socket.unpack_sockaddr_in(EM::get_sockname(@server))
     end
 
     def stop_server
@@ -54,7 +76,7 @@ class Watcher
 
     def kill
         if @io
-            Process.kill 'TERM', @io.pid 
+            Process.kill 9, @io.pid
             Process.waitpid @io.pid
             @io = nil
         end
@@ -67,7 +89,7 @@ class Watcher
     end
 
     def path_changed path
-        @handler.call path
+        @handler.call path, true
     end
 
     def listener_died