vagrant-gatling-rsync 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: d981cfb603768c87a0a307b415b32acf4fdacc27
+  data.tar.gz: 7517d55a2e0be0a9d6fc04c7ae60c15262ebc82d
+SHA512:
+  metadata.gz: 6e0938d9cc6f4e67da39c1865c004b348a7ef03451ed8b93086af4dff9706101dee20a5313c4006153803588edd4def025b538a2c0fb9e1779404ad880dfb992
+  data.tar.gz: 176593bea18b5cc114ebd0f9ce944ccfe7c7438beb128ea738e770056ee38cc0301ede1641bfcacbcf339ea599008cc8d159f26396dda8a55cdac798692fdce6

data/.gitignore

@@ -0,0 +1,21 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+*.swp
+*.swo
+example/files/*
+example/vagrant/.vagrant

data/.ruby-version

@@ -0,0 +1 @@
+2.0.0-p353

data/CHANGELOG.txt

@@ -0,0 +1,16 @@
+## 0.0.1 (March 23, 2014)
+
+Initial release.
+
+FEATURES:
+
+- Implement an rb-fsevent adapter for Mac.
+- Implement an rb-inotify adapter for Linux.
+- Add the `vagrant gatling-rsync-auto` command.
+- Implement the first draft of a config option to allow changing the latency.
+
+## Backlog
+
+- Test and release the Windows adapter.
+- Allow configuring and running the rsync daemon to avoid SSH overhead.
+- Get latency validation working.

data/Gemfile

@@ -0,0 +1,14 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in vagrant-gatling-rsync.gemspec
+gemspec
+
+group :development do
+  gem "vagrant", :git => "https://github.com/mitchellh/vagrant.git", :tag => 'v1.5.1'
+  gem "pry"
+end
+
+group :plugins do
+  gem "vagrant-gatling-rsync", path: "."
+end
+

data/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2014 Steven Merrill
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,73 @@
+# vagrant-gatling-rsync
+
+An rsync watcher for Vagrant 1.5+ that uses fewer host resources at the
+potential cost of more rsync actions.
+
+## Authors
+
+Steven Merrill (@stevenmerrill) originally had the idea to tap into rb-fsevent
+and rb-inotify to more efficiently rsync files.
+
+Doug Marcey (@dougmarcey) provided considerable guidance in the implementation
+of the coalescing functionality and wrote the initial sketch of the Linux and
+Windows adapters.
+
+## Why "gatling"?
+
+The gatling gun was the first gun capable of firing continuously.
+
+## This plugin
+
+The built-in rsync-auto plugin sometimes uses a lot of CPU and disk I/O when
+it starts up on very large rsynced directories. This plugin is designed to
+work well with such large rsynced folders.
+
+The rsync-auto command that ships with Vagrant 1.5 uses the listen gem. The
+Listen gem is quite thorough - it uses Celluloid to spin up an actor system
+and it checks file contents on OS X to ensure that running "touch" on a file
+(to do a write but not update its content) will not fire the rsync command.
+
+The downside of using Listen is that it takes a large amount of host resources
+to monitor large directory structures. This gem works well with to monitor
+directories hierarchies with 10,000-100,000 files.
+
+This gem's implementation is much closer to the underlying fsevent or inotify
+APIs, which allows for higher performance.
+
+## Event coalescing
+
+This plugin also coalesces events for you. The default latency is 1.5 seconds.
+(This will be a configurable option in future versions.) If you specify a
+latency of two seconds, this plugin will not fire a `vagrant rsync` until two
+contiguous seconds without file events have passed. This will delay rsyncs from
+happening if many writes are happening on the host (during a `make` or a
+`git clone`, for example) until the activity has leveled off.
+
+## Working with this plugin
+
+Add the following information to the Vagrantfile to set the coalescing
+threshold in seconds. If you do not set it, it will default to 1.5.
+
+You will also need to have at least one synced folder set to type "rsync"
+to use the plugin.
+
+```ruby
+VAGRANTFILE_API_VERSION = "2"
+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+  config.vm.box = "hashicorp/precise64"
+
+  config.vm.synced_folder "../files", "/opt/vagrant/rsynced_folder", type: "rsync",
+    rsync__exclude: [".git/", ".idea/"]
+
+  # Configure the window for gatling to coalesce writes.
+  config.gatling.latency = 2.5
+end
+```
+
+With the Vagrantfile configured in this fashion, you can run the following
+command to sync files.
+
+```bash
+vagrant gatling-rsync-auto
+```

data/Rakefile

@@ -0,0 +1 @@
+require "bundler/gem_tasks"

data/example/files/.gitkeep

@@ -0,0 +1 @@
+This file only exists to keep this directory in git.

data/example/vagrant/Vagrantfile

@@ -0,0 +1,19 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
+VAGRANTFILE_API_VERSION = "2"
+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+  config.vm.box = "hashicorp/precise64"
+
+  # Share an additional folder to the guest VM. The first argument is
+  # the path on the host to the actual folder. The second argument is
+  # the path on the guest to mount the folder. And the optional third
+  # argument is a set of non-required options.
+  config.vm.synced_folder "../files", "/opt/vagrant/rsynced_folder", type: "rsync",
+    rsync__exclude: [".git/", ".idea/"]
+
+  # Configure the window for gatling to coalesce writes.
+  config.gatling.latency = 2.5
+end

data/lib/vagrant-gatling-rsync/command/rsync_auto.rb

@@ -0,0 +1,130 @@
+require "log4r"
+require "optparse"
+
+require "vagrant"
+
+module VagrantPlugins
+  module GatlingRsync
+    class GatlingRsyncAuto < Vagrant.plugin(2, :command)
+      include Vagrant::Action::Builtin::MixinSyncedFolders
+
+      def self.synopsis
+        "syncs rsync synced folders when folders change"
+      end
+
+      def execute
+        @logger = Log4r::Logger.new("vagrant::commands::gatling-rsync-auto")
+
+        opts = OptionParser.new do |o|
+          o.banner = "Usage: vagrant gatling-rsync-auto [vm-name]"
+          o.separator ""
+        end
+
+        # Parse the options and return if we don't have any target.
+        argv = parse_options(opts)
+        return if !argv
+
+        latency = nil
+
+        # Build up the paths that we need to listen to.
+        paths = {}
+        ignores = []
+        with_target_vms(argv) do |machine|
+          latency = machine.config.gatling.latency
+
+          folders = synced_folders(machine)[:rsync]
+          next if !folders || folders.empty?
+
+          folders.each do |id, folder_opts|
+            # If we marked this folder to not auto sync, then
+            # don't do it.
+            next if folder_opts.has_key?(:auto) && !folder_opts[:auto]
+
+            hostpath = folder_opts[:hostpath]
+            hostpath = File.expand_path(hostpath, machine.env.root_path)
+            paths[hostpath] ||= []
+            paths[hostpath] << {
+              id: id,
+              machine: machine,
+              opts: folder_opts,
+            }
+
+            if folder_opts[:exclude]
+              Array(folder_opts[:exclude]).each do |pattern|
+                ignores << VagrantPlugins::SyncedFolderRSync::RsyncHelper.exclude_to_regexp(hostpath, pattern.to_s)
+              end
+            end
+          end
+        end
+
+        # Output to the user what paths we'll be watching
+        paths.keys.sort.each do |path|
+          paths[path].each do |path_opts|
+            path_opts[:machine].ui.info(I18n.t(
+              "vagrant.rsync_auto_path",
+              path: path.to_s,
+            ))
+          end
+        end
+
+        @logger.info("Listening to paths: #{paths.keys.sort.inspect}")
+        @logger.info("Ignoring #{ignores.length} paths:")
+        ignores.each do |ignore|
+          @logger.info("  -- #{ignore.to_s}")
+        end
+
+        case RUBY_PLATFORM
+        when /darwin/
+          ListenOSX.new(paths, ignores, latency, @logger, self.method(:callback)).run
+        when /linux/
+          ListenLinux.new(paths, ignores, latency, @logger, self.method(:callback)).run
+        else
+          # @TODO: Raise this earlier?
+          raise Errors::OnlyOSXLinuxSupportError
+        end
+
+        0
+      end
+
+      # This callback gets called when any directory changes.
+      def callback(paths, ignores, modified)
+        @logger.info("File change callback called!")
+        @logger.info("  - Paths: #{paths.inspect}")
+        @logger.info("  - Ignores: #{ignores.inspect}")
+        @logger.info("  - Modified: #{modified.inspect}")
+
+        tosync = []
+        paths.each do |hostpath, folders|
+          # Find out if this path should be synced
+          found = catch(:done) do
+            modified.each do |changed|
+              match = nil
+              ignores.each do |ignore|
+                next unless match.nil?
+                match = ignore.match(changed)
+              end
+
+              next unless match.nil?
+              throw :done, true if changed.start_with?(hostpath)
+            end
+
+            # Make sure to return false if all else fails so that we
+            # don't sync to this machine.
+            false
+          end
+
+          # If it should be synced, store it for later
+          tosync << folders if found
+        end
+
+        # Sync all the folders that need to be synced
+        tosync.each do |folders|
+          folders.each do |opts|
+            ssh_info = opts[:machine].ssh_info
+            VagrantPlugins::SyncedFolderRSync::RsyncHelper.rsync_single(opts[:machine], ssh_info, opts[:opts])
+          end
+        end
+      end
+    end
+  end
+end

data/lib/vagrant-gatling-rsync/config.rb

@@ -0,0 +1,36 @@
+require "vagrant"
+
+module VagrantPlugins
+  module GatlingRsync
+    class Config < Vagrant.plugin("2", :config)
+      attr_accessor :latency
+
+      def initialize
+        @latency = UNSET_VALUE
+      end
+
+      def finalize!
+        @latency = 1.5 if @latency == UNSET_VALUE
+      end
+
+      # @TODO: This does not appear to be called.
+      def validate(machine)
+        errors = _detected_errors
+
+        if @latency == UNSET_VALUE
+          return
+        elsif not @latency.is_a? Numeric
+          @latency = 1.5
+          # @TODO: Translate.
+          errors << "The latency must be set to a number. Substituting 1.5 as a value."
+        elsif @latency < 0.2
+          @latency = 0.2
+          # @TODO: Translate.
+          errors << "The latency may not be below 0.2 seconds."
+        end
+
+        { "gatling" => errors }
+      end
+    end
+  end
+end

data/lib/vagrant-gatling-rsync/errors.rb

@@ -0,0 +1,19 @@
+require "vagrant"
+
+module VagrantPlugins
+  module GatlingRsync
+    module Errors
+      class VagrantGatlingRsyncError < Vagrant::Errors::VagrantError
+        error_namespace("vagrant_gatling_rsync.errors")
+      end
+
+      class OnlyOSXLinuxSupportError < VagrantGatlingRsyncError
+        error_key(:only_osx_linux_support)
+      end
+
+      class Vagrant15RequiredError < VagrantGatlingRsyncError
+        error_key(:vagrant_15_required)
+      end
+    end
+  end
+end

data/lib/vagrant-gatling-rsync/listen/listenlinux.rb

@@ -0,0 +1,44 @@
+require "rb-inotify"
+
+module VagrantPlugins
+  module GatlingRsync
+    class ListenLinux
+      def initialize(paths, ignores, latency, logger, callback)
+        @paths = paths
+        @ignores = ignores
+        @latency = latency
+        @logger = logger
+        @callback = callback
+      end
+
+      def run
+        @logger.info("Listening via: rb-inotify on Linux.")
+
+        notifier = INotify::Notifier.new
+        @paths.keys.each do |path|
+          notifier.watch(path, :modify, :create, :delete, :recursive) {}
+        end
+
+        while true do
+          directories = Set.new
+          begin
+            while true do
+              events = []
+              events = Timeout::timeout(@latency) {
+                notifier.read_events
+              }
+              events.each { |e| directories << e.absolute_name }
+            end
+          rescue Timeout::Error
+            @logger.info("Breaking out of the loop at #{Time.now.to_s}.")
+          end
+
+          @logger.info("Detected changes to #{directories.inspect}.") unless directories.empty?
+
+          @callback.call(@paths, @ignores, directories) unless directories.empty?
+        end
+      end
+    end
+  end
+end
+

data/lib/vagrant-gatling-rsync/listen/listenosx.rb

@@ -0,0 +1,53 @@
+require "rb-fsevent"
+
+module VagrantPlugins
+  module GatlingRsync
+    class ListenOSX
+      def initialize(paths, ignores, latency, logger, callback)
+        @paths = paths
+        @ignores = ignores
+        @latency = latency
+        @options = {
+          # We set this to a small value to ensure that we can coalesce the
+          # events together to prevent rsyncing too often under heavy write
+          # load.
+          :latency => 0.1,
+          :no_defer => false,
+        }
+        @logger = logger
+        @callback = callback
+      end
+
+      def run
+        @logger.info("Listening via: rb-fsevent on Mac OS X.")
+        changes = Queue.new
+
+        fsevent = FSEvent.new
+        fsevent.watch @paths.keys, @options do |directories|
+          directories.each { |d| changes << d }
+        end
+        Thread.new { fsevent.run }
+
+        while true do
+          directories = Set.new
+          begin
+            while true do
+              @logger.info("Starting the timeout at #{Time.now.to_s}.")
+              change = Timeout::timeout(@latency) {
+                changes.pop
+              }
+              directories << change unless change.nil?
+            end
+          rescue Timeout::Error
+            @logger.info("Breaking out of the loop at #{Time.now.to_s}.")
+          end
+
+          @logger.info("Detected changes to #{directories.inspect}.") unless directories.empty?
+
+          @callback.call(@paths, @ignores, directories) unless directories.empty?
+        end
+      end
+    end
+  end
+end
+

data/lib/vagrant-gatling-rsync/listen/listenwindows.rb

@@ -0,0 +1,33 @@
+# @TODO: Note that this is entirely untested and not yet implemented.
+
+require "timeout"
+require "thread"
+
+paths = {"/path/to/a/directory" => {}}
+latency = 2
+
+monitor = WDM::Monitor.new
+changes = Queue.new
+paths.keys.each do |path|
+  monitor.watch_recursively(path) { |change| changes << change }
+end
+Thread.new { monitor.run! }
+
+while true do
+  directories = Set.new
+  begin
+    while true do
+      change = Timeout::timeout(latency) {
+        changes.pop
+      }
+      directories << change.path
+    end
+  rescue Timeout::Error
+    @logger.info("Breaking out of the loop at #{Time.now.to_s}.")
+  end
+
+  @logger.info(directories.inspect) unless directories.empty?
+
+  #@callback.call(paths, ignores, directories) unless directories.empty?
+end
+

data/lib/vagrant-gatling-rsync/plugin.rb

@@ -0,0 +1,40 @@
+begin
+  require "vagrant"
+rescue LoadError
+  raise "The Vagrant gatling rsync plugin must be run within Vagrant."
+end
+
+# This is a sanity check to make sure no one is attempting to install
+# this into an early Vagrant version.
+if Vagrant::VERSION < "1.5.0"
+  raise Errors::Vagrant15RequiredError
+end
+
+module VagrantPlugins
+  module GatlingRsync
+    class Plugin < Vagrant.plugin("2")
+      name "Gatling Rsync"
+      description <<-DESC
+      Rsync large project directories to your Vagrant VM without using many resources on the host.
+      DESC
+
+      # This initializes the internationalization strings.
+      def self.setup_i18n
+        I18n.load_path << File.expand_path("locales/en.yml", GatlingRsync.source_root)
+        I18n.reload!
+      end
+
+      command "gatling-rsync-auto" do
+        setup_i18n
+
+        require_relative "command/rsync_auto"
+        GatlingRsyncAuto
+      end
+
+      config "gatling" do
+        require_relative "config"
+        Config
+      end
+    end
+  end
+end

data/lib/vagrant-gatling-rsync/version.rb

@@ -0,0 +1,5 @@
+module VagrantPlugins
+  module GatlingRsync
+    VERSION = "0.0.1"
+  end
+end

data/lib/vagrant-gatling-rsync.rb

@@ -0,0 +1,25 @@
+# This file is required because Vagrant's plugin system expects
+# a eponymous ruby file matching the rubygem.
+#
+# So this gem is called 'vagrant-gatling-rsync' and thus vagrant tries
+# to require "vagrant-gatling-rsync"
+
+require "vagrant-gatling-rsync/plugin"
+
+require "pathname"
+
+module VagrantPlugins
+  module GatlingRsync
+    lib_path = Pathname.new(File.expand_path("../vagrant-gatling-rsync", __FILE__))
+    autoload :Errors, lib_path.join("errors")
+    autoload :ListenOSX, lib_path.join("listen/listenosx")
+    autoload :ListenLinux, lib_path.join("listen/listenlinux")
+
+    # This returns the path to the source of this plugin.
+    #
+    # @return [Pathname]
+    def self.source_root
+      @source_root ||= Pathname.new(File.expand_path("../../", __FILE__))
+    end
+  end
+end

data/locales/en.yml

@@ -0,0 +1,8 @@
+en:
+  vagrant_gatling_rsync:
+    errors:
+      only_osx_linux_support: |-
+        This plugin currently only supports Max OS X and Linux hosts.
+      vagrant_15_required: |-
+        This plugin requires Vagrant 1.5 or newer to function.
+

data/vagrant-gatling-rsync.gemspec

@@ -0,0 +1,25 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'vagrant-gatling-rsync/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "vagrant-gatling-rsync"
+  spec.version       = VagrantPlugins::GatlingRsync::VERSION
+  spec.authors       = ["Steven Merrill"]
+  spec.email         = ["steven.merrill@gmail.com"]
+  spec.summary       = %q{A lighter-weight Vagrant plugin for watching and rsyncing directories.}
+  spec.description   = %q{The gatling-rsync plugin run on Mac (and soon on Linux) and is far less CPU-intensive than the built-in 'vagrant rsync-auto' plugin, at the cost of possibly rsyncing more often.}
+  spec.homepage      = ""
+  spec.license       = "MIT"
+
+  # @TODO: Remove example files from the built gem.
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.5"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "pry"
+end

metadata

@@ -0,0 +1,108 @@
+--- !ruby/object:Gem::Specification
+name: vagrant-gatling-rsync
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- Steven Merrill
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2014-03-24 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.5'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.5'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: pry
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: The gatling-rsync plugin run on Mac (and soon on Linux) and is far less
+  CPU-intensive than the built-in 'vagrant rsync-auto' plugin, at the cost of possibly
+  rsyncing more often.
+email:
+- steven.merrill@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- .ruby-version
+- CHANGELOG.txt
+- Gemfile
+- LICENSE
+- README.md
+- Rakefile
+- example/files/.gitkeep
+- example/vagrant/Vagrantfile
+- lib/vagrant-gatling-rsync.rb
+- lib/vagrant-gatling-rsync/command/rsync_auto.rb
+- lib/vagrant-gatling-rsync/config.rb
+- lib/vagrant-gatling-rsync/errors.rb
+- lib/vagrant-gatling-rsync/listen/listenlinux.rb
+- lib/vagrant-gatling-rsync/listen/listenosx.rb
+- lib/vagrant-gatling-rsync/listen/listenwindows.rb
+- lib/vagrant-gatling-rsync/plugin.rb
+- lib/vagrant-gatling-rsync/version.rb
+- locales/en.yml
+- vagrant-gatling-rsync.gemspec
+homepage: ''
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.0.14
+signing_key: 
+specification_version: 4
+summary: A lighter-weight Vagrant plugin for watching and rsyncing directories.
+test_files: []