logstash-input-eventlog 0.1.0 → 3.0.0

checksums.yaml

@@ -1,15 +1,7 @@
 ---
-!binary "U0hBMQ==":
-  metadata.gz: !binary |-
-    ZDBjNWRiYmZlZThlYzFkMjJiYTIxMDk2YzU2OTgwNTRkOTc3ZjA5MA==
-  data.tar.gz: !binary |-
-    MDQ5ZDVmNzc0MDU2NDcxYmNmZmY0ZmI3MzcxNzNjYmRlZTkxMDBiZA==
+SHA1:
+  metadata.gz: d383f435401333671e4c67f299fba906218b5e62
+  data.tar.gz: 618396042270ad53179b98b6f6bd63df0c6d5b5d
 SHA512:
-  metadata.gz: !binary |-
-    NDUxYWE5MDJkZDc2Nzg3NjQ0YjJhODNhMTY5ZWViMzhhMWQzZGJjNzgzZDcy
-    YWIzODQ4OGVlMmFhOWM1YTBkZmIxOTEyNWM3MDgwMzFkNDIzZmQ3ZjAyNjMy
-    NTE1ODQ2YTlmMTYxMjBkMzllNjJiNjlhYmRiOTg1YzcyYWJhZDA=
-  data.tar.gz: !binary |-
-    MjgwMjFlMGNjYTcyZDRjODBkOWRlMDAzZjlkODliOGQyMDJkOWEzYzA3Yzk1
-    Yzc2MGY4NGVlYmNkMjg5MTU4ZTkwMGY2N2U4ZjM2MTk4MTYzZmY4MTMxMDU1
-    M2Y1YmFhMmNlYzRkZGMzNzBkYTU1ZDZlMjAyNDQwOTg1MTJjZjU=
+  metadata.gz: 0f48194926185e234ffe916d8bdf78471c0f5b1736a930bb81bc92c75d1000e9ea8b024a32e0bb3757cbba4856bf5e650afa015f49dc23e1ab758480df8defc2
+  data.tar.gz: 26da2254059f6cbf31f2cc16baf1e356eb3979620b89d628616fc4fdbcf18aee73353d4c581e9f40c404a3ccf361ef8d75d09124920c28c1aea2d60651c629ff

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+## 2.0.0
+ - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully, 
+   instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
+ - Dependency on logstash-core update to 2.0
+

data/CONTRIBUTORS

@@ -0,0 +1,19 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Bernd Ahlers (bernd)
+* Colin Surprenant (colinsurprenant)
+* Drew Rogers (drogerschariot)
+* Greg Mefford (GregMefford)
+* James Turnbull (jamtur01)
+* Jordan Sissel (jordansissel)
+* Kurt Hurtado (kurtado)
+* Pier-Hugues Pellerin (ph)
+* Richard Pijnenburg (electrical)
+* Suyog Rao (suyograo)
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -1,4 +1,2 @@
-source 'http://rubygems.org'
-gem 'rake'
-gem 'gem_publisher'
-gem 'archive-tar-minitar'
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2012-2014 Elasticsearch <http://www.elasticsearch.org>
+Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.

data/NOTICE.TXT

@@ -0,0 +1,5 @@
+Elasticsearch
+Copyright 2012-2015 Elasticsearch
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -0,0 +1,89 @@
+# Logstash Plugin
+
+[![Build
+Status](http://build-eu-00.elastic.co/view/LS%20Plugins/view/LS%20Inputs/job/logstash-plugin-input-eventlog-unit/badge/icon)](http://build-eu-00.elastic.co/view/LS%20Plugins/view/LS%20Inputs/job/logstash-plugin-input-eventlog-unit/)
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+bin/plugin install --no-verify
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/inputs/eventlog.rb

@@ -2,12 +2,13 @@
 require "logstash/inputs/base"
 require "logstash/namespace"
 require "logstash/timestamp"
-require "socket"
+require "win32/eventlog"
+require "stud/interval"
 
-# This input will pull events from a (http://msdn.microsoft.com/en-us/library/windows/desktop/bb309026%28v=vs.85%29.aspx)[Windows Event Log].
+# This input will pull events from a http://msdn.microsoft.com/en-us/library/windows/desktop/bb309026%28v=vs.85%29.aspx[Windows Event Log].
 #
 # To collect Events from the System Event Log, use a config like:
-#
+# [source,ruby]
 #     input {
 #       eventlog {
 #         type  => 'Win32-EventLog'
@@ -17,113 +18,75 @@ require "socket"
 class LogStash::Inputs::EventLog < LogStash::Inputs::Base
 
   config_name "eventlog"
-  milestone 2
 
   default :codec, "plain"
 
   # Event Log Name
-  config :logfile, :validate => :array, :default => [ "Application", "Security", "System" ]
+  # System and Security may require that privileges are given to the user running logstash.
+  # see more at: https://social.technet.microsoft.com/forums/windowsserver/en-US/d2f813db-6142-4b5b-8d86-253ebb740473/easy-way-to-read-security-log
+  config :logfile, :validate => :string, :validate => [ "Application", "Security", "System" ], :default => "Application"
+
+  # How frequently should tail check for new event logs in ms (default: 1 second)
+  config :interval, :validate => :number, :default => 1000
 
   public
   def register
 
     # wrap specified logfiles in suitable OR statements
-    @logfiles = @logfile.join("' OR TargetInstance.LogFile = '")
-
     @hostname = Socket.gethostname
-    @logger.info("Registering input eventlog://#{@hostname}/#{@logfile}")
+    @logger.info("Opening eventlog #{@logfile}")
 
-    if RUBY_PLATFORM == "java"
-      require "jruby-win32ole"
-    else
-      require "win32ole"
+    begin
+      @eventlog = Win32::EventLog.open(@logfile)
+    rescue SystemCallError => e
+      if e.errno == 1314 # ERROR_PRIVILEGE_NOT_HELD
+        @logger.fatal("No privilege held to open logfile", :logfile => @logfile)
+      end
+      raise
     end
   end # def register
 
   public
   def run(queue)
-    @wmi = WIN32OLE.connect("winmgmts://")
 
-    wmi_query = "Select * from __InstanceCreationEvent Where TargetInstance ISA 'Win32_NTLogEvent' And (TargetInstance.LogFile = '#{@logfiles}')"
-
-    begin
-      @logger.debug("Tailing Windows Event Log '#{@logfile}'")
-
-      events = @wmi.ExecNotificationQuery(wmi_query)
-
-      while
-        notification = events.NextEvent
-        event = notification.TargetInstance
-
-        timestamp = to_timestamp(event.TimeGenerated)
-
-        e = LogStash::Event.new(
-          "host" => @hostname,
-          "path" => @logfile,
-          "type" => @type,
-          LogStash::Event::TIMESTAMP => timestamp
-        )
-
-        %w{Category CategoryString ComputerName EventCode EventIdentifier
-            EventType Logfile Message RecordNumber SourceName
-            TimeGenerated TimeWritten Type User
-        }.each{
-            |property| e[property] = event.send property
-        }
-
-        if RUBY_PLATFORM == "java"
-          # unwrap jruby-win32ole racob data
-          e["InsertionStrings"] = unwrap_racob_variant_array(event.InsertionStrings)
-          data = unwrap_racob_variant_array(event.Data)
-          # Data is an array of signed shorts, so convert to bytes and pack a string
-          e["Data"] = data.map{|byte| (byte > 0) ? byte : 256 + byte}.pack("c*")
-        else
-          # win32-ole data does not need to be unwrapped
-          e["InsertionStrings"] = event.InsertionStrings
-          e["Data"] = event.Data
-        end
-
-        e["message"] = event.Message
-
-        decorate(e)
-        queue << e
-
-      end # while
-
-    rescue Exception => ex
-      @logger.error("Windows Event Log error: #{ex}\n#{ex.backtrace}")
-      sleep 1
-      retry
-    end # rescue
-
-  end # def run
-
-  private
-  def unwrap_racob_variant_array(variants)
-    variants ||= []
-    variants.map {|v| (v.respond_to? :getValue) ? v.getValue : v}
-  end # def unwrap_racob_variant_array
+    @logger.debug("Tailing Windows Event Log '#{@logfile}'")
+
+    old_total = @eventlog.total_records()
+    flags     = Win32::EventLog::FORWARDS_READ | Win32::EventLog::SEEK_READ
+    rec_num   = @eventlog.read_last_event.record_number
+
+    while !stop?
+      new_total = @eventlog.total_records()
+      if new_total != old_total
+        rec_num = @eventlog.oldest_record_number() if @eventlog.full?
+        @eventlog.read(flags, rec_num).each { |log| e = process(log); decorate(e); queue << e }
+        old_total = new_total
+        rec_num   = @eventlog.read_last_event.record_number + 1
+      end
+      Stud.stoppable_sleep(@interval/1000.0) { stop? }
+    end
+  end
 
-  # the event log timestamp is a utc string in the following format: yyyymmddHHMMSS.xxxxxx±UUU
-  # http://technet.microsoft.com/en-us/library/ee198928.aspx
   private
-  def to_timestamp(wmi_time)
-    result = ""
-    # parse the utc date string
-    /(?<w_date>\d{8})(?<w_time>\d{6})\.\d{6}(?<w_sign>[\+-])(?<w_diff>\d{3})/ =~ wmi_time
-    result = "#{w_date}T#{w_time}#{w_sign}"
-    # the offset is represented by the difference, in minutes,
-    # between the local time zone and Greenwich Mean Time (GMT).
-    if w_diff.to_i > 0
-      # calculate the timezone offset in hours and minutes
-      h_offset = w_diff.to_i / 60
-      m_offset = w_diff.to_i - (h_offset * 60)
-      result.concat("%02d%02d" % [h_offset, m_offset])
-    else
-      result.concat("0000")
-    end
+  def process(log)
+
+    LogStash::Event.new(
+      "host"             => @hostname,
+      "Logfile"          => @logfile,
+      "message"          => log["description"].strip,
+      "Category"         => log["category"],
+      "ComputerName"     => log["computer"],
+      "EventIdentifier"  => log["event_id"],
+      "EventType"        => log["event_type"],
+      "RecordNumber"     => log["record_number"],
+      "SourceName"       => log["source"],
+      "TimeGenerated"    => log["time_generated"],
+      "TimeWritten"      => log["time_written"],
+      "Type"             => log["event_type"],
+      "User"             => log["user"],
+      "InsertionStrings" => log["string_inserts"]
+    )
+  end # def run
 
-    return LogStash::Timestamp.new(DateTime.strptime(result, "%Y%m%dT%H%M%S%z").to_time)
-  end
 end # class LogStash::Inputs::EventLog
 

data/logstash-input-eventlog.gemspec

@@ -1,32 +1,30 @@
 Gem::Specification.new do |s|
 
   s.name            = 'logstash-input-eventlog'
-  s.version         = '0.1.0'
+  s.version         = '3.0.0'
   s.licenses        = ['Apache License (2.0)']
   s.summary         = "This input will pull events from a Windows Event Log"
-  s.description     = "This input will pull events from a Windows Event Log"
-  s.authors         = ["Elasticsearch"]
-  s.email           = 'richard.pijnenburg@elasticsearch.com'
-  s.homepage        = "http://logstash.net/"
+  s.description     = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
+  s.authors         = ["Elastic"]
+  s.email           = 'info@elastic.co'
+  s.homepage        = "http://www.elastic.co/guide/en/logstash/current/index.html"
   s.require_paths = ["lib"]
 
   # Files
-  s.files = `git ls-files`.split($\)+::Dir.glob('vendor/*')
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
 
   # Tests
   s.test_files = s.files.grep(%r{^(test|spec|features)/})
 
   # Special flag to let us know this is actually a logstash plugin
-  s.metadata = { "logstash_plugin" => "true", "group" => "input" }
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "input" }
 
   # Gem dependencies
-  s.add_runtime_dependency 'logstash', '>= 1.4.0', '< 2.0.0'
+  s.add_runtime_dependency "logstash-core", ">= 2.0.0.beta2", "< 3.0.0"
 
   s.add_runtime_dependency 'logstash-codec-plain'
 
-  if RUBY_PLATFORM == 'java'
-    s.platform = RUBY_PLATFORM
-    s.add_runtime_dependency "jruby-win32ole"                   #(unknown license)
-  end
+  s.add_runtime_dependency "win32-eventlog", "~> 0.6.5"  # Artistic 2.0
+  s.add_runtime_dependency "stud", "~> 0.0.22"   # Apache 2.0
+  s.add_development_dependency 'logstash-devutils'
 end
-

data/spec/inputs/eventlog_spec.rb

@@ -1,5 +1,8 @@
-require 'spec_helper'
+require "logstash/devutils/rspec/spec_helper"
 require 'logstash/inputs/eventlog'
 
-describe LogStash::Inputs::EventLog do
+describe LogStash::Inputs::EventLog, :windows => true do
+  it_behaves_like "an interruptible input plugin" do
+    let(:config) { { "logfile" => "Application", "interval" => 10000000 } }
+  end
 end

metadata

@@ -1,89 +1,130 @@
 --- !ruby/object:Gem::Specification
 name: logstash-input-eventlog
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 3.0.0
 platform: ruby
 authors:
-- Elasticsearch
-autorequire: 
+- Elastic
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2014-11-05 00:00:00.000000000 Z
+date: 2015-11-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: logstash
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
-        version: 1.4.0
+        version: 2.0.0.beta2
     - - <
       - !ruby/object:Gem::Version
-        version: 2.0.0
-  type: :runtime
+        version: 3.0.0
+  name: logstash-core
   prerelease: false
+  type: :runtime
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
-        version: 1.4.0
+        version: 2.0.0.beta2
     - - <
       - !ruby/object:Gem::Version
-        version: 2.0.0
+        version: 3.0.0
 - !ruby/object:Gem::Dependency
-  name: logstash-codec-plain
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-codec-plain
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.6.5
+  name: win32-eventlog
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.6.5
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.0.22
+  name: stud
+  prerelease: false
   type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.0.22
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-devutils
   prerelease: false
+  type: :development
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-description: This input will pull events from a Windows Event Log
-email: richard.pijnenburg@elasticsearch.com
+description: This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program
+email: info@elastic.co
 executables: []
 extensions: []
 extra_rdoc_files: []
 files:
-- .gitignore
+- CHANGELOG.md
+- CONTRIBUTORS
 - Gemfile
-- Gemfile.bak
 - LICENSE
-- Rakefile
+- NOTICE.TXT
+- README.md
 - lib/logstash/inputs/eventlog.rb
 - logstash-input-eventlog.gemspec
-- rakelib/publish.rake
-- rakelib/vendor.rake
 - spec/inputs/eventlog_spec.rb
-homepage: http://logstash.net/
+homepage: http://www.elastic.co/guide/en/logstash/current/index.html
 licenses:
 - Apache License (2.0)
 metadata:
   logstash_plugin: 'true'
-  group: input
-post_install_message: 
+  logstash_group: input
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ! '>='
+  - - '>='
     - !ruby/object:Gem::Version
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ! '>='
+  - - '>='
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.4.1
-signing_key: 
+rubyforge_project:
+rubygems_version: 2.4.8
+signing_key:
 specification_version: 4
 summary: This input will pull events from a Windows Event Log
 test_files:

data/.gitignore

@@ -1,4 +0,0 @@
-*.gem
-Gemfile.lock
-.bundle
-vendor

data/Gemfile.bak

@@ -1,4 +0,0 @@
-source 'http://rubygems.org'
-gem 'rake'
-gem 'gem_publisher'
-gem 'archive-tar-minitar'

data/Rakefile

@@ -1,6 +0,0 @@
-@files=[]
-
-task :default do
-  system("rake -T")
-end
-

data/rakelib/publish.rake

@@ -1,9 +0,0 @@
-require "gem_publisher"
-
-desc "Publish gem to RubyGems.org"
-task :publish_gem do |t|
-  gem_file = Dir.glob(File.expand_path('../*.gemspec',File.dirname(__FILE__))).first
-  gem = GemPublisher.publish_if_updated(gem_file, :rubygems)
-  puts "Published #{gem}" if gem
-end
-

data/rakelib/vendor.rake

@@ -1,169 +0,0 @@
-require "net/http"
-require "uri"
-require "digest/sha1"
-
-def vendor(*args)
-  return File.join("vendor", *args)
-end
-
-directory "vendor/" => ["vendor"] do |task, args|
-  mkdir task.name
-end
-
-def fetch(url, sha1, output)
-
-  puts "Downloading #{url}"
-  actual_sha1 = download(url, output)
-
-  if actual_sha1 != sha1
-    fail "SHA1 does not match (expected '#{sha1}' but got '#{actual_sha1}')"
-  end
-end # def fetch
-
-def file_fetch(url, sha1)
-  filename = File.basename( URI(url).path )
-  output = "vendor/#{filename}"
-  task output => [ "vendor/" ] do
-    begin
-      actual_sha1 = file_sha1(output)
-      if actual_sha1 != sha1
-        fetch(url, sha1, output)
-      end
-    rescue Errno::ENOENT
-      fetch(url, sha1, output)
-    end
-  end.invoke
-
-  return output
-end
-
-def file_sha1(path)
-  digest = Digest::SHA1.new
-  fd = File.new(path, "r")
-  while true
-    begin
-      digest << fd.sysread(16384)
-    rescue EOFError
-      break
-    end
-  end
-  return digest.hexdigest
-ensure
-  fd.close if fd
-end
-
-def download(url, output)
-  uri = URI(url)
-  digest = Digest::SHA1.new
-  tmp = "#{output}.tmp"
-  Net::HTTP.start(uri.host, uri.port, :use_ssl => (uri.scheme == "https")) do |http|
-    request = Net::HTTP::Get.new(uri.path)
-    http.request(request) do |response|
-      fail "HTTP fetch failed for #{url}. #{response}" if [200, 301].include?(response.code)
-      size = (response["content-length"].to_i || -1).to_f
-      count = 0
-      File.open(tmp, "w") do |fd|
-        response.read_body do |chunk|
-          fd.write(chunk)
-          digest << chunk
-          if size > 0 && $stdout.tty?
-            count += chunk.bytesize
-            $stdout.write(sprintf("\r%0.2f%%", count/size * 100))
-          end
-        end
-      end
-      $stdout.write("\r      \r") if $stdout.tty?
-    end
-  end
-
-  File.rename(tmp, output)
-
-  return digest.hexdigest
-rescue SocketError => e
-  puts "Failure while downloading #{url}: #{e}"
-  raise
-ensure
-  File.unlink(tmp) if File.exist?(tmp)
-end # def download
-
-def untar(tarball, &block)
-  require "archive/tar/minitar"
-  tgz = Zlib::GzipReader.new(File.open(tarball))
-  # Pull out typesdb
-  tar = Archive::Tar::Minitar::Input.open(tgz)
-  tar.each do |entry|
-    path = block.call(entry)
-    next if path.nil?
-    parent = File.dirname(path)
-    
-    mkdir_p parent unless File.directory?(parent)
-
-    # Skip this file if the output file is the same size
-    if entry.directory?
-      mkdir path unless File.directory?(path)
-    else
-      entry_mode = entry.instance_eval { @mode } & 0777
-      if File.exists?(path)
-        stat = File.stat(path)
-        # TODO(sissel): Submit a patch to archive-tar-minitar upstream to
-        # expose headers in the entry.
-        entry_size = entry.instance_eval { @size }
-        # If file sizes are same, skip writing.
-        next if stat.size == entry_size && (stat.mode & 0777) == entry_mode
-      end
-      puts "Extracting #{entry.full_name} from #{tarball} #{entry_mode.to_s(8)}"
-      File.open(path, "w") do |fd|
-        # eof? check lets us skip empty files. Necessary because the API provided by
-        # Archive::Tar::Minitar::Reader::EntryStream only mostly acts like an
-        # IO object. Something about empty files in this EntryStream causes
-        # IO.copy_stream to throw "can't convert nil into String" on JRuby
-        # TODO(sissel): File a bug about this.
-        while !entry.eof?
-          chunk = entry.read(16384)
-          fd.write(chunk)
-        end
-          #IO.copy_stream(entry, fd)
-      end
-      File.chmod(entry_mode, path)
-    end
-  end
-  tar.close
-  File.unlink(tarball) if File.file?(tarball)
-end # def untar
-
-def ungz(file)
-
-  outpath = file.gsub('.gz', '')
-  tgz = Zlib::GzipReader.new(File.open(file))
-  begin
-    File.open(outpath, "w") do |out|
-      IO::copy_stream(tgz, out)
-    end
-    File.unlink(file)
-  rescue
-    File.unlink(outpath) if File.file?(outpath)
-   raise
-  end
-  tgz.close
-end
-
-desc "Process any vendor files required for this plugin"
-task "vendor" do |task, args|
-
-  @files.each do |file| 
-    download = file_fetch(file['url'], file['sha1'])
-    if download =~ /.tar.gz/
-      prefix = download.gsub('.tar.gz', '').gsub('vendor/', '')
-      untar(download) do |entry|
-        if !file['files'].nil?
-          next unless file['files'].include?(entry.full_name.gsub(prefix, ''))
-          out = entry.full_name.split("/").last
-        end
-        File.join('vendor', out)
-      end
-    elsif download =~ /.gz/
-      ungz(download)
-    end
-  end
-
-end