logstash-lib 1.3.2

data/docs/tutorials/zeromq.md

@@ -0,0 +1,118 @@
+---
+title: ZeroMQ - logstash
+layout: content_right
+---
+
+*ZeroMQ support in Logstash is currently in an experimental phase. As such, parts of this document are subject to change.*
+
+# ZeroMQ
+Simply put ZeroMQ (0mq) is a socket on steroids. This makes it a perfect compliment to Logstash - a pipe on steroids.
+
+ZeroMQ allows you to easily create sockets of various types for moving data around. These sockets are refered to in ZeroMQ by the behavior of each side of the socket pair:
+
+* PUSH/PULL
+* REQ/REP
+* PUB/SUB
+* ROUTER/DEALER
+
+There is also a `PAIR` socket type as well.
+
+Additionally, the socket type is independent of the connection method. A PUB/SUB socket pair could have the SUB side of the socket be a listener and the PUB side a connecting client. This makes it very easy to fit ZeroMQ into various firewalled architectures.
+
+Note that this is not a full-fledged tutorial on ZeroMQ. It is a tutorial on how Logstash uses ZeroMQ.
+
+# ZeroMQ and logstash
+In the spirit of ZeroMQ, Logstash takes these socket type pairs and uses them to create topologies with some very simply rules that make usage very easy to understand:
+
+* The receiving end of a socket pair is always a logstash input
+* The sending end of a socket pair is always a logstash output
+* By default, inputs `bind`/listen and outputs `connect`
+* Logstash refers to the socket pairs as topologies and mirrors the naming scheme from ZeroMQ
+* By default, ZeroMQ inputs listen on all interfaces on port 2120, ZeroMQ outputs connect to `localhost` on port 2120
+
+The currently understood Logstash topologies for ZeroMQ inputs and outputs are:
+
+* `pushpull`
+* `pubsub`
+* `pair`
+
+We have found from various discussions that these three topologies will cover most of user's needs. We hope to expose the full span of ZeroMQ socket types as time goes on.
+
+By keeping the options simple, this allows you to get started VERY easily with what are normally complex message flows. No more confusion over `exchanges` and `queues` and `brokers`. If you need to add fanout capability to your flow, you can simply use the following configs:
+
+* _node agent lives at 192.168.1.2_
+* _indexer agent lives at 192.168.1.1_
+
+    # Node agent config
+    input { stdin { type => "test-stdin-input" } }
+    output { zeromq { topology => "pubsub" address => "tcp://192.168.1.1.:2120" } }
+
+    # Indexer agent config
+    input { zeromq { topology => "pubsub" } }
+    output { stdout { debug => true }}
+
+If for some reason you need connections to initiate from the indexer because of firewall rules:
+
+    # Node agent config - now listening on all interfaces port 2120
+    input { stdin { type => "test-stdin-input" } }
+    output { zeromq { topology => "pubsub" address => "tcp://*.:2120" mode => "server" } }
+
+    # Indexer agent config
+    input { zeromq { topology => "pubsub" address => "tcp://192.168.1.2" mode => "client" } }
+    output { stdout { debug => true }}
+
+As stated above, by default `inputs` always start as listeners and `outputs` always start as initiators. Please don't confuse what happens once the socket is connect with the direction of the connection. ZeroMQ separates connection from topology. In the second case of the above configs, once the two sockets are connected, regardless of who initiated the connection, the message flow itself is absolute. The indexer is reading events from the node.
+
+# Which topology to use
+The choice of topology can be broken down very easily based on need
+
+## one to one
+Use `pair` topology. On the output side, specify the ipaddress and port of the input side.
+
+## broadcast
+Use `pubsub`
+If you need to broadcast ALL messages to multiple hosts that each need to see all events, use `pubsub`. Note that all events are broadcast to all subscribers. When using `pubsub` you might also want to investigate the `topic` configuration option which allows subscribers to see only a subset of messages.
+
+## Filter workers
+Use `pushpull`
+In `pushpull`, ZeroMQ automatically load balances to all connected peers. This means that no peer sees the same message as any other peer.
+
+# What's with the address format?
+ZeroMQ supports multiple types of transports:
+
+* inproc:// (unsupported by logstash due to threading)
+* tcp:// (exactly what it sounds like)
+* ipc:// (probably useless in logstash)
+* pgm:// and epgm:// (a multicast format - only usable with PUB and SUB socket types)
+
+For pretty much all cases, you'll be using `tcp://` transports with Logstash.
+
+## Topic - applies to `pubsub`
+This opt mimics the routing keys functionality in AMQP. Imagine you have a network of receivers but only a subset of the messages need to be seen by a subset of the hosts. You can use this option as a routing key to facilite that:
+
+    # This output is a PUB
+    output {
+    zeromq { topology => "pubsub" topic => "logs.production.%{host}" }
+    }
+
+    # This input is a SUB
+    # I only care about db1 logs
+    input { zeromq { type => "db1logs" address => "tcp://<ipaddress>:2120" topic => "logs.production.db1"}}
+
+One thing important to note about 0mq PUBSUB and topics is that all filtering is done on the subscriber side. The subscriber will get ALL messages but discard any that don't match the topic.
+
+Also important to note is that 0mq doesn't do topic in the same sense as an AMQP broker might. When a SUB socket gets a message, it compares the first bytes of the message against the topic. However, this isn't always flexible depending on the format of your message. The common practice then, is to send a 0mq multipart message and make the first part the topic. The next parts become the actual message body.
+
+This is approach is how logstash handles this. When using PUBSUB, Logstash will send a multipart message where the first part is the name of the topic and the second part is the event. This is important to know if you are sending to a SUB input from sources other than Logstash.
+
+# sockopts
+Sockopts is not you choosing between blue or black socks. ZeroMQ supports setting various flags or options on sockets. In the interest of minimizing configuration syntax, these are _hidden_ behind a logstash configuration element called `sockopts`. You probably won't need to tune these for most cases. If you do need to tune them, you'll probably set the following:
+
+## ZMQ::HWM - sets the high water mark
+The high water mark is the maximum number of messages a given socket pair can have in its internal queue. Use this to throttle essentially.
+
+## ZMQ::SWAP_SIZE
+TODO
+
+## ZMQ::IDENTITY
+TODO

data/extract_services.rb

@@ -0,0 +1,29 @@
+# Extract META-INFO/services/* files from jars
+#
+require "optparse"
+
+output = nil
+
+flags = OptionParser.new do |opts|
+  opts.on("-o", "--output DIR",
+          "Where to write the merged META-INF/services/* files") do |dir|
+    output = dir
+  end
+end
+
+flags.parse!(ARGV)
+
+ARGV.each do |jar|
+  # Find any files matching /META-INF/services/* in any jar given on the
+  # command line.
+  # Append all file content to the output directory with the same file name
+  # as is in the jar.
+  glob = "file:///#{File.expand_path(jar)}!/META-INF/services/*"
+  Dir.glob(glob).each do |service|
+    name = File.basename(service)
+    File.open(File.join(output, name), "a") do |fd|
+      puts "Adding #{name} from #{File.basename(jar)}"
+      fd.write(File.read(service))
+    end
+  end
+end

data/gembag.rb

@@ -0,0 +1,64 @@
+#!/usr/bin/env ruby
+
+require "rbconfig"
+
+rubyabi = RbConfig::CONFIG["ruby_version"]
+target = "#{Dir.pwd}/vendor/bundle"
+gemdir = "#{target}/#{RUBY_ENGINE}/#{rubyabi}/"
+ENV["GEM_HOME"] = gemdir
+ENV["GEM_PATH"] = ""
+
+require "rubygems/specification"
+require "rubygems/commands/install_command"
+
+def install_gem(name, requirement, target)
+  puts "Fetching and installing gem: #{name} (#{requirement})"
+
+  installer = Gem::Commands::InstallCommand.new
+  installer.options[:generate_rdoc] = false
+  installer.options[:generate_ri] = false
+  installer.options[:version] = requirement
+  installer.options[:args] = [name]
+  installer.options[:install_dir] = target
+ 
+  # ruby 2.0.0 / rubygems 2.x; disable documentation generation
+  installer.options[:document] = []
+  begin
+    installer.execute
+  rescue Gem::SystemExitException => e
+    if e.exit_code != 0
+      puts "Installation of #{name} failed"
+      raise
+    end
+  end
+end # def install_gem
+
+# Ensure bundler is available.
+begin
+  gem("bundler", ">=1.3.5")
+rescue Gem::LoadError => e
+  install_gem("bundler", ">= 1.3.5", ENV["GEM_HOME"])
+end
+
+require "bundler/cli"
+
+# Monkeypatch bundler to write a .lock file specific to the version of ruby.
+# This keeps MRI/JRuby/RBX from conflicting over the Gemfile.lock updates
+module Bundler
+  module SharedHelpers
+    def default_lockfile
+      ruby = "#{RUBY_ENGINE}-#{RbConfig::CONFIG["ruby_version"]}"
+      return Pathname.new("#{default_gemfile}.#{ruby}.lock")
+    end
+  end
+end
+
+if RUBY_ENGINE == "rbx"
+  begin
+    gem("rubysl")
+  rescue Gem::LoadError => e
+    install_gem("rubysl", ">= 0", ENV["GEM_HOME"])
+  end
+end
+
+Bundler::CLI.start(["install", "--gemfile=tools/Gemfile", "--path", target, "--clean", "--jobs", "4"])

data/lib/logstash-event.rb

@@ -0,0 +1,2 @@
+# encoding: utf-8
+require "logstash/event"

data/lib/logstash.rb

@@ -0,0 +1,4 @@
+# encoding: utf-8
+require "logstash/agent"
+require "logstash/event"
+require "logstash/namespace"

data/lib/logstash/JRUBY-6970-openssl.rb

@@ -0,0 +1,22 @@
+# encoding: utf-8
+# TODO(sissel): require "openssl" takes *ages* from the logstash jar
+# TODO(sissel): monkeypatch Kernel.require to apply this monkeypatch only after
+# a 'require "openssl" has occurred.
+class OpenSSL::SSL::SSLContext
+  alias_method :ca_path_JRUBY_6970=, :ca_path=
+  alias_method :ca_file_JRUBY_6970=, :ca_file=
+
+  def ca_file=(arg)
+    if arg =~ /^jar:file:\//
+      return ca_file_JRUBY_6970=(arg.gsub(/^jar:/, ""))
+    end
+    return ca_file_JRUBY_6970=(arg)
+  end
+
+  def ca_path=(arg)
+    if arg =~ /^jar:file:\//
+      return ca_path_JRUBY_6970=(arg.gsub(/^jar:/, ""))
+    end
+    return ca_path_JRUBY_6970=(arg)
+  end
+end

data/lib/logstash/JRUBY-6970.rb

@@ -0,0 +1,102 @@
+# encoding: utf-8
+# Monkeypatch for JRUBY-6970
+module Kernel
+  alias_method :require_JRUBY_6970_hack, :require
+
+  def require(path)
+    old_load_path = nil
+
+    if path =~ /^jar:file:.+!.+/
+      path = path.gsub(/^jar:/, "")
+      puts "JRUBY-6970: require(#{path})" if ENV["REQUIRE_DEBUG"] == "1"
+    end
+
+    # Work around slow openssl load times in flatjar. (LOGSTASH-1223)
+    # I don't know why this works, I don't care either. This problem only
+    # exists in the 'jar' builds of logstash which are going to be going away
+    # soon in favor of the much-better tarball/zip releases!
+    if __FILE__ =~ /^(?:jar:)?file:.+!.+/ && path == "openssl"
+      # Loading shared/jruby-openssl first seems to make openssl load faster
+      # I have no idea. Computers.
+      require_JRUBY_6970_hack "shared/jruby-openssl"
+      return require_JRUBY_6970_hack "openssl"
+    end
+
+    # JRUBY-7065
+    path = File.expand_path(path) if path.include?("/../")
+    rc = require_JRUBY_6970_hack(path)
+
+    # Only monkeypatch openssl after it's been loaded.
+    if path == "openssl"
+      require "logstash/JRUBY-6970-openssl"
+    end
+    return rc
+  end
+end
+
+# Work around for a bug in File.expand_path that doesn't account for resources
+# in jar paths.
+#
+# Should solve this error:
+#   Exception in thread "LogStash::Runner" org.jruby.exceptions.RaiseException:
+#   (Errno::ENOENT) file:/home/jls/projects/logstash/build/data/unicode.data
+class File
+  class << self
+    alias_method :expand_path_JRUBY_6970, :expand_path
+
+    def expand_path(path, dir=nil)
+      #p :expand_path => [path, dir]
+      if path =~ /(jar:)?file:\/.*\.jar!/
+        #p :expand_path_path => [path, dir]
+        jar, resource = path.split("!", 2)
+        #p :expand_path => [jar, resource]
+        if resource.nil? || resource == ""
+          # Nothing after the "!", nothing special to handle.
+          return expand_path_JRUBY_6970(path, dir)
+        else
+          resource = expand_path_JRUBY_6970(resource, dir)
+          return fix_jar_path(jar, resource)
+        end
+      elsif dir =~ /(jar:)?file:\/.*\.jar!/
+        jar, dir = dir.split("!", 2)
+        if dir.empty?
+          # sometimes the original dir is just 'file:/foo.jar!'
+          return File.join("#{jar}!", path) 
+        end
+        dir = expand_path_JRUBY_6970(path, dir)
+        return fix_jar_path(jar, dir)
+      else
+        return expand_path_JRUBY_6970(path, dir)
+      end
+    end
+  end
+
+  protected
+
+  def self.fix_jar_path(jar, resource)
+    # TODO(sissel): use LogStash::Util::UNAME
+    if RbConfig::CONFIG["host_os"] == "mswin32"
+      # 'expand_path' on "/" will return "C:/" on windows.
+      # So like.. we don't want that because technically this
+      # is the root of the jar, not of a disk.
+      #puts :fix_jar_path => ["#{jar}!#{resource.gsub(/^[A-Za-z]:/, "")}"]
+      return "#{jar}!#{resource.gsub(/^[A-Za-z]:/, "")}"
+    else
+      return "#{jar}!#{resource}"
+    end
+  end
+end
+
+class Dir
+  class << self
+    alias_method :glob_JRUBY_6970_hack, :glob
+    def glob(path, flags=nil)
+      if path =~ /^jar:file:/
+        # Strip leading 'jar:' (LOGSTASH-1316)
+        return glob_JRUBY_6970_hack(path.gsub(/^jar:/, ""))
+      else
+        return glob_JRUBY_6970_hack(path)
+      end
+    end
+  end
+end

data/lib/logstash/agent.rb

@@ -0,0 +1,305 @@
+# encoding: utf-8
+require "clamp" # gem 'clamp'
+require "logstash/errors"
+require "i18n"
+
+class LogStash::Agent < Clamp::Command
+  option ["-f", "--config"], "CONFIG_PATH",
+    I18n.t("logstash.agent.flag.config"),
+    :attribute_name => :config_path
+
+  option "-e", "CONFIG_STRING",
+    I18n.t("logstash.agent.flag.config-string"),
+    :attribute_name => :config_string
+
+  option ["-w", "--filterworkers"], "COUNT",
+    I18n.t("logstash.agent.flag.filterworkers"),
+    :attribute_name => :filter_workers, :default => 1, &:to_i
+
+  option "--watchdog-timeout", "SECONDS", 
+    I18n.t("logstash.agent.flag.watchdog-timeout"),
+    :default => 10, &:to_f
+
+  option ["-l", "--log"], "FILE",
+    I18n.t("logstash.agent.flag.log"),
+    :attribute_name => :log_file
+
+  # Old support for the '-v' flag'
+  option "-v", :flag, 
+    I18n.t("logstash.agent.flag.verbosity"),
+    :attribute_name => :verbosity, :multivalued => true
+
+  option "--quiet", :flag, I18n.t("logstash.agent.flag.quiet")
+  option "--verbose", :flag, I18n.t("logstash.agent.flag.verbose")
+  option "--debug", :flag, I18n.t("logstash.agent.flag.debug")
+
+  option ["-V", "--version"], :flag,
+    I18n.t("logstash.agent.flag.version")
+
+  option ["-p", "--pluginpath"] , "PATH",
+    I18n.t("logstash.agent.flag.pluginpath"),
+    :multivalued => true,
+    :attribute_name => :plugin_paths
+
+  option ["-t", "--configtest"], :flag,
+    I18n.t("logstash.agent.flag.configtest"),
+    :attribute_name => :config_test
+
+  # Emit a warning message.
+  def warn(message)
+    # For now, all warnings are fatal.
+    raise LogStash::ConfigurationError, message
+  end # def warn
+
+  # Emit a failure message and abort.
+  def fail(message)
+    raise LogStash::ConfigurationError, message
+  end # def fail
+
+  def report(message)
+    # Print to stdout just in case we're logging to a file
+    puts message
+    @logger.log(message) if log_file
+  end
+
+  # Run the agent. This method is invoked after clamp parses the
+  # flags given to this program.
+  def execute
+    require "logstash/pipeline"
+    require "cabin" # gem 'cabin'
+    require "logstash/plugin"
+    @logger = Cabin::Channel.get(LogStash)
+
+    if version?
+      show_version
+      return 0
+    end
+
+    # temporarily send logs to stdout as well if a --log is specified
+    # and stdout appears to be a tty
+    show_startup_errors = log_file && STDOUT.tty?
+
+    if show_startup_errors
+      stdout_logs = @logger.subscribe(STDOUT)
+    end
+    configure
+
+    # You must specify a config_string or config_path
+    if config_string.nil? && config_path.nil?
+      fail(help + "\n" + I18n.t("logstash.agent.missing-configuration"))
+    end
+
+    if @config_path
+      @config_string = load_config(@config_path)
+    else
+      # include a default stdin input if no inputs given
+      if @config_string !~ /input *{/
+        @config_string += "input { stdin { type => stdin } }"
+      end
+      # include a default stdout output if no outputs given
+      if @config_string !~ /output *{/
+        @config_string += "output { stdout { codec => rubydebug } }"
+      end
+    end
+
+    begin
+      pipeline = LogStash::Pipeline.new(@config_string)
+    rescue LoadError => e
+      fail("Configuration problem.")
+    end
+
+    # Stop now if we are only asking for a config test.
+    if config_test?
+      report "Configuration OK"
+      return
+    end
+
+    # Make SIGINT shutdown the pipeline.
+    trap_id = Stud::trap("INT") do
+      @logger.warn(I18n.t("logstash.agent.interrupted"))
+      pipeline.shutdown
+    end
+
+    Stud::trap("HUP") do
+      @logger.info(I18n.t("logstash.agent.sighup"))
+      configure_logging(log_file)
+    end
+
+    pipeline.configure("filter-workers", filter_workers)
+
+    @logger.unsubscribe(stdout_logs) if show_startup_errors
+
+    # TODO(sissel): Get pipeline completion status.
+    pipeline.run
+    return 0
+  rescue LogStash::ConfigurationError => e
+    @logger.unsubscribe(stdout_logs) if show_startup_errors
+    report I18n.t("logstash.agent.error", :error => e)
+    return 1
+  rescue => e
+    @logger.unsubscribe(stdout_logs) if show_startup_errors
+    report I18n.t("oops", :error => e)
+    report e.backtrace if @logger.debug? || $DEBUGLIST.include?("stacktrace")
+    return 1
+  ensure
+    @log_fd.close if @log_fd
+    Stud::untrap("INT", trap_id) unless trap_id.nil?
+  end # def execute
+
+  def show_version
+    show_version_logstash
+
+    if [:info, :debug].include?(verbosity?) || debug? || verbose?
+      show_version_ruby
+
+      if RUBY_PLATFORM == "java"
+        show_version_java
+        show_version_elasticsearch
+      end
+
+      if [:debug].include?(verbosity?) || debug?
+        show_gems 
+      end
+    end
+  end # def show_version
+
+  def show_version_logstash
+    require "logstash/version"
+    puts "logstash #{LOGSTASH_VERSION}"
+  end # def show_version_logstash
+
+  def show_version_ruby
+    puts RUBY_DESCRIPTION
+  end # def show_version_ruby
+
+  def show_version_elasticsearch
+    # Not running in the,jar? assume elasticsearch jars are
+    # in ../../vendor/jar/...
+    if __FILE__ !~ /^(?:jar:)?file:/
+      jarpath = File.join(File.dirname(__FILE__), "../../vendor/jar/elasticsearch*/lib/*.jar")
+      Dir.glob(jarpath).each do |jar|
+        require jar
+      end
+    end
+
+    $stdout.write("Elasticsearch: ");
+    org.elasticsearch.Version::main([])
+  end # def show_version_elasticsearch
+
+  def show_version_java
+    properties = java.lang.System.getProperties
+    puts "java #{properties["java.version"]} (#{properties["java.vendor"]})"
+    puts "jvm #{properties["java.vm.name"]} / #{properties["java.vm.version"]}"
+  end # def show_version_java
+
+  def show_gems
+    require "rubygems"
+    Gem::Specification.each do |spec|
+      puts "gem #{spec.name} #{spec.version}"
+    end
+  end # def show_gems
+
+  # Do any start-time configuration.
+  #
+  # Log file stuff, plugin path checking, etc.
+  def configure
+    configure_logging(log_file)
+    configure_plugin_path(plugin_paths) if !plugin_paths.nil?
+  end # def configure
+
+  # Point logging at a specific path.
+  def configure_logging(path)
+    # Set with the -v (or -vv...) flag
+    if quiet?
+      @logger.level = :error
+    elsif verbose?
+      @logger.level = :info
+    elsif debug?
+      @logger.level = :debug
+    else
+      # Old support for the -v and -vv stuff.
+      if verbosity? && verbosity?.any?
+        # this is an array with length of how many times the flag is given
+        if verbosity?.length == 1
+          @logger.warn("The -v flag is deprecated and will be removed in a future release. You should use --verbose instead.")
+          @logger.level = :info
+        else
+          @logger.warn("The -vv flag is deprecated and will be removed in a future release. You should use --debug instead.")
+          @logger.level = :debug
+        end
+      else
+        @logger.level = :warn
+      end
+
+    end
+
+    if log_file
+      # TODO(sissel): Implement file output/rotation in Cabin.
+      # TODO(sissel): Catch exceptions, report sane errors.
+      begin
+        @log_fd.close if @log_fd
+        @log_fd = File.new(path, "a")
+      rescue => e
+        fail(I18n.t("logstash.agent.configuration.log_file_failed",
+                    :path => path, :error => e))
+      end
+
+      puts "Sending logstash logs to #{path}."
+      @logger.unsubscribe(@logger_subscription) if @logger_subscription
+      @logger_subscription = @logger.subscribe(@log_fd)
+    else
+      @logger.subscribe(STDOUT)
+    end
+
+    # TODO(sissel): redirect stdout/stderr to the log as well
+    # http://jira.codehaus.org/browse/JRUBY-7003
+  end # def configure_logging
+
+  # Validate and add any paths to the list of locations
+  # logstash will look to find plugins.
+  def configure_plugin_path(paths)
+    # Append any plugin paths to the ruby search path
+    paths.each do |path|
+      # Verify the path exists
+      if !Dir.exists?(path)
+        warn(I18n.t("logstash.agent.configuration.plugin_path_missing",
+                    :path => path))
+
+      end
+
+      # TODO(sissel): Verify the path looks like the correct form.
+      # aka, there must be file in path/logstash/{filters,inputs,outputs}/*.rb
+      plugin_glob = File.join(path, "logstash", "{inputs,filters,outputs}", "*.rb")
+      if Dir.glob(plugin_glob).empty?
+        warn(I18n.t("logstash.agent.configuration.no_plugins_found",
+                    :path => path, :plugin_glob => plugin_glob))
+      end
+
+      # We push plugin paths to the front of the LOAD_PATH so that folks
+      # can override any core logstash plugins if they need to.
+      @logger.debug("Adding plugin path", :path => path)
+      $LOAD_PATH.unshift(path)
+    end
+  end # def configure_plugin_path
+
+  def load_config(path)
+    path = File.join(path, "*") if File.directory?(path)
+
+    if Dir.glob(path).length == 0
+      fail(I18n.t("logstash.agent.configuration.file-not-found", :path => path))
+    end
+
+    config = ""
+    Dir.glob(path).sort.each do |file|
+      next unless File.file?(file)
+      if file.match(/~$/)
+        @logger.debug("NOT reading config file because it is a temp file", :file => file)
+        next
+      end
+      @logger.debug("Reading config file", :file => file)
+      config << File.read(file) + "\n"
+    end
+    return config
+  end # def load_config
+
+end # class LogStash::Agent