logstash-lib 1.3.2

data/README.md

@@ -0,0 +1,120 @@
+# logstash
+
+[![Build Status](https://secure.travis-ci.org/logstash/logstash.png)](http://travis-ci.org/logstash/logstash)
+
+logstash is a tool for managing events and logs. You can use it to collect
+logs, parse them, and store them for later use (like, for searching). Speaking
+of searching, logstash comes with a web interface for searching and drilling
+into all of your logs.
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you
+are pretty much free to use it however you want in whatever way.
+
+For more info, see <http://logstash.net/>
+
+## Need Help?
+
+Need help? Try #logstash on freenode irc or the logstash-users@googlegroups.com
+mailing list.
+
+You can also find documentation on the <http://logstash.net> site.
+
+## Developing
+
+If you don't have JRuby already (or don't use rvm, rbenv, etc), you can have `bin/logstash` fetch it for you by setting `USE_JRUBY`:
+
+    USE_JRUBY=1 bin/logstash ...
+
+Otherwise, here's how to get started with rvm: 
+
+    # Install JRuby with rvm
+    rvm install jruby-1.7.8
+    rvm use jruby-1.7.8
+
+Now install dependencies:
+
+    # Install logstash ruby dependencies
+    bin/logstash deps
+
+Other commands:
+
+    # to use logstash gems or libraries in irb, use the following
+    # this gets you an 'irb' shell with logstash's environment
+    bin/logstash irb
+
+    # or use irb from the jar
+    java -jar logstash-<version>-monolithic.jar irb
+
+    # Run logstash
+    bin/logstash agent [options]
+    
+    # If running bin/logstash agent yields complaints about log4j/other things
+    # This will download the elasticsearch jars so logstash can use them.
+    make vendor-elasticsearch
+
+## Testing
+
+There are a few ways to run the tests. For development, using `bin/logstash
+rspec <some spec>` will suffice:
+
+    % bin/logstash rspec spec/filters/grok.rb 
+    ...................
+
+    Finished in 0.123 seconds
+    19 examples, 0 failures
+
+Alternately, if you have just built the flatjar, you can run the tests
+specifically on those like so:
+
+    make flatjar-test
+
+If you want to run all the tests from source (not compiled jar), do:
+
+    make test
+
+Finally, like 'bin/logstash rspec' above, you can invoke the jar to run a
+specific test like so:
+
+    % java -jar logstash.jar rspec spec/filters/grok.rb
+    ...................
+
+    Finished in 0.346 seconds
+    19 examples, 0 failures
+
+## Building
+
+Building is not required. You are highly recommended to download the releases
+we provide from the logstash site!
+
+If you want to build the jar yourself, run:
+
+    make flatjar
+
+To update a flat jar previously built with 'make flatjar', run:
+
+    make update-flatjar
+
+
+You can build rpms and debs, if you need those. Building rpms requires you have [fpm](github.com/jordansissel/fpm), then do this:
+
+    make package
+
+## Project Principles
+
+* Community: If a newbie has a bad time, it's a bug.
+* Software: Make it work, then make it right, then make it fast.
+* Technology: If it doesn't do a thing today, we can make it do it tomorrow.
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports,
+complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and
+maintainers or community members  saying "send patches or die" - you will not
+see that here.
+
+It is more important to me that you are able to contribute.
+
+For more information about contributing, see the
+[CONTRIBUTING](CONTRIBUTING.md) file.

data/STYLE.md

@@ -0,0 +1,96 @@
+# Style Guide 
+
+Rather than write a full style guide, please follow by examples you see in the
+code. If you send me a patch, I will not reject it for style reasons (but I
+will fix it before it gets committed).
+
+## Logging
+
+We support logging structured data, so please do that.
+
+Rather than this:
+
+    @logger.info("Some error occured in request #{request} on input #{input} from client #{ip}")
+
+Do this:
+    
+    @logger.info("Some error occured in this request", :request => request, :input => input, :client => ip)
+
+## Code Style
+
+* comment everything you can think of.
+* indentation: 2 spaces
+* between methods: 1 line
+* sort your requires
+* long lines should wrap at 80 characters. If you wrap at an operator ('or',
+  '+', etc) start the next line with that operator.
+* parentheses on function definitions/calls
+* explicit is better than implicit
+  * implicit returns are forbidden except in the case of a single expression 
+
+The point is consistency and documentation. If you see inconsistencies, let me
+know, and I'll fix them :)
+
+Short example:
+
+      require "something from a gem" # from gem 'thing'
+
+      # some documentation about this class
+      class Foo < Bar
+        # some documentation about this function
+        def somefunc(arg1, arg2, arg3)
+          # comment
+          puts "Hello"
+          if (some_long_condition \
+              or some_other_condition)
+            puts "World"
+          end # if <very short description>
+
+          # Long lines should wrap and start with an operator if possible.
+          foo = some + long + formula + thing \
+                + stuff + bar;
+
+          # Function calls, when wrapping, should align to the '(' where reasonable.
+          some_function_call(arg1, arg2, arg3, some_long_thing,
+                             alignment_here, arg6)
+          # If it seems unreasonable, wrap and indent 4 spaces.
+          some_really_long_function_call_blah_blah_blah(arg1,
+              arg2, arg3, arg4)
+
+          # indent the 'when' inside a 'case'.
+          case foo
+            when "bar"
+              puts "Hello world"
+            when /testing/
+              puts "testing
+            else
+              puts "I got nothin'"
+          end # case foo
+            
+        end # def somefunc
+      end # class Foo
+
+## Specific cases
+
+### Hash Syntax
+
+Use of the "hash colon" syntax (ruby 1.9) is not accepted.
+
+    # This is NOT good.
+    { foo: "bar" }
+
+    # This is good.
+    { :foo => "bar" }
+
+### String#[]
+
+String#[] with one numeric argument must not be used due to bugs and
+inconsistencies between ruby versions.
+
+    str = "foo"
+
+    # This is NOT good
+    str[0]
+
+    # This is good.
+    str[0, 1]

data/bin/logstash

@@ -0,0 +1,37 @@
+#!/bin/sh
+# Run logstash from source
+#
+# This is most useful when done from a git checkout.
+#
+# Usage:
+#     bin/logstash <command> [arguments]
+#
+# See 'bin/logstash help' for a list of commands.
+#
+# NOTE: One extra command is available 'deps'
+# The 'deps' command will install dependencies for logstash.
+#
+# If you do not have ruby installed, you can set "USE_JRUBY=1"
+# in your environment and this script will download and use
+# a release of JRuby for you.
+
+# Defaults you can override with environment variables
+LS_HEAP_SIZE="${LS_HEAP_SIZE:=500m}"
+
+basedir=$(cd `dirname $0`/..; pwd)
+. ${basedir}/bin/logstash.lib.sh
+
+setup
+
+case $1 in
+  deps) install_deps ;;
+  -*) 
+    # is the first argument a flag? If so, assume 'agent'
+    program="$basedir/lib/logstash/runner.rb"
+    exec $RUBYCMD "$program" agent "$@"
+    ;;
+  *)
+    program="$basedir/lib/logstash/runner.rb"
+    exec $RUBYCMD "$program" "$@"
+    ;;
+esac

data/bin/logstash-test

@@ -0,0 +1,4 @@
+#!/bin/sh
+
+basedir=$(cd `dirname $0`/..; pwd)
+exec $basedir/bin/logstash rspec "$@"

data/bin/logstash-web

@@ -0,0 +1,4 @@
+#!/bin/sh
+
+basedir=$(cd `dirname $0`/..; pwd)
+exec $basedir/bin/logstash web "$@"

data/bin/logstash.lib.sh

@@ -0,0 +1,78 @@
+basedir=$(cd `dirname $0`/..; pwd)
+
+setup_ruby() {
+  # Verify ruby works
+  if ! ruby -e 'puts "HURRAY"' 2> /dev/null | grep -q "HURRAY" ; then
+    echo "No ruby program found. Cannot start."
+    exit 1
+  fi
+
+  eval $(ruby -rrbconfig -e 'puts "RUBYVER=#{RbConfig::CONFIG["ruby_version"]}"; puts "RUBY=#{RUBY_ENGINE}"')
+  RUBYCMD="ruby"
+}
+
+setup_java() {
+  if [ -z "$JAVA_HOME/bin/java" ] ; then
+    JAVA="$JAVA_HOME/bin/java"
+  else
+    JAVA=$(which java)
+  fi
+
+  if [ ! -x "$JAVA" ] ; then
+    echo "Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME."
+    exit 1
+  fi
+
+
+  JAVA_OPTS="$JAVA_OPTS -Xmx${LS_HEAP_SIZE}"
+  JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
+  JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC"
+  JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"
+
+  JAVA_OPTS="$JAVA_OPTS -XX:CMSInitiatingOccupancyFraction=75"
+  JAVA_OPTS="$JAVA_OPTS -XX:+UseCMSInitiatingOccupancyOnly"
+
+  if [ ! -z "$LS_USE_GC_LOGGING" ] ; then
+    JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
+    JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
+    JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
+    JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
+    JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
+    JAVA_OPTS="$JAVA_OPTS -Xloggc:./logstash-gc.log"
+    echo "Writing garbage collection logs to ./logstash-gc.log"
+  fi
+} 
+
+setup_vendored_jruby() {
+  RUBYVER=1.9
+  RUBY=jruby
+
+  setup_java
+
+  RUBYCMD="$JAVA $JAVA_OPTS -jar $basedir/vendor/jar/jruby-complete-*.jar"
+}
+
+setup() {
+  if [ -z "$USE_JRUBY" -a \( -d "$basedir/.git" -o ! -z "$USE_RUBY" \) ] ; then
+    setup_ruby
+    if [ "$RUBY" = "jruby" ] ; then
+      setup_java
+      export JAVA_OPTS
+    fi
+  else
+    setup_vendored_jruby
+  fi
+  export GEM_HOME="$basedir/vendor/bundle/${RUBY}/${RUBYVER}"
+  export GEM_PATH=
+  export RUBYLIB="$basedir/lib"
+}
+
+install_deps() {
+  if [ -f "$basedir/logstash.gemspec" ] ; then
+    program="$basedir/gembag.rb"
+    set -- "$basedir/logstash.gemspec"
+    exec $RUBYCMD "$basedir/gembag.rb" "$@"
+  else
+    echo "Cannot install dependencies; missing logstash.gemspec. This 'deps' command only works from a logstash git clone."
+  fi
+}

data/bot/check_pull_changelog.rb

@@ -0,0 +1,89 @@
+require "octokit"
+##
+# This script will validate that any pull request submitted against a github 
+# repository will contains changes to CHANGELOG file.
+#
+# If not the case, an helpful text will be commented on the pull request
+# If ok, a thanksful message will be commented also containing a @mention to 
+# acts as a trigger for review notification by a human.
+## 
+
+
+@bot="" # Put here your bot github username
+@password="" # Put here your bot github password
+
+@repository="logstash/logstash"
+@mention="@jordansissel"
+
+@missing_changelog_message = <<MISSING_CHANGELOG
+Hello, I'm #{@bot}, I'm here to help you accomplish your pull request submission quest
+
+You still need to accomplish these tasks:
+
+* Please add a changelog information
+
+Also note that your pull request name will appears in the details section 
+of the release notes, so please make it clear
+MISSING_CHANGELOG
+
+@ok_changelog_message = <<OK_CHANGELOG
+You successfully completed the pre-requisite quest (aka updating CHANGELOG)
+
+Also note that your pull request name will appears in the details section 
+of the release notes, so please make it clear, if not already done.
+
+#{@mention} Dear master, would you please have a look to this humble request
+OK_CHANGELOG
+
+#Connect to Github
+@client=Octokit::Client.new(:login => @bot, :password => @password)
+
+
+#For each open pull
+Octokit.pull_requests(@repository).each do |pull|
+  #Get botComment
+  botComment = nil
+  @client.issue_comments(@repository, pull.number, {
+    :sort => "created",
+    :direction => "desc"
+  }).each do |comment|
+    if comment.user.login == @bot
+      botComment = comment
+      break
+    end
+  end
+
+  if !botComment.nil? and botComment.body.start_with?("[BOT-OK]")
+    #Pull already validated by bot, nothing to do
+    puts "Pull request #{pull.number}, already ok for bot"
+  else
+    #Firt encounter, or previous [BOT-WARN] status
+    #Check for changelog
+    warnOnMissingChangeLog = true
+    @client.pull_request_files(@repository, pull.number).each do |changedFile|
+      if changedFile.filename  == "CHANGELOG"
+        if changedFile.additions.to_i > 0
+          #Changelog looks good
+          warnOnMissingChangeLog = false
+        else
+          #No additions, means crazy deletion
+          warnOnMissingChangeLog = true
+        end
+      end
+    end
+    if warnOnMissingChangeLog
+      if botComment.nil?
+        puts "Pull request #{pull.number}, adding bot warning"
+        @client.add_comment(@repository, pull.number, "[BOT-WARN] #{@missing_changelog_message}")
+      else
+        puts "Pull request #{pull.number}, already warned, no changes yet"
+      end
+    else
+      if !botComment.nil?
+        @client.delete_comment(@repository,botComment.id)
+      end
+      puts "Pull request #{pull.number}, adding bot ok"
+      @client.add_comment(@repository, pull.number, "[BOT-OK] #{@ok_changelog_message}")
+    end
+  end
+end

data/docs/configuration.md

@@ -0,0 +1,260 @@
+---
+title: Configuration Language - logstash
+layout: content_right
+---
+# LogStash Config Language
+
+The logstash config language aims to be simple.
+
+There's 3 main sections: inputs, filters, outputs. Each section has
+configurations for each plugin available in that section.
+
+Example:
+
+    # This is a comment. You should use comments to describe
+    # parts of your configuration.
+    input {
+      ...
+    }
+
+    filter {
+      ...
+    }
+
+    output {
+      ...
+    }
+
+## Filters and Ordering
+
+For a given event, are applied in the order of appearance in the
+configuration file.
+
+## Comments
+
+Comments are as in ruby, perl, and python. Starts with a '#' character. Example:
+
+    # this is a comment
+
+    input { # comments can appear at the end of a line, too
+      # ...
+    }
+
+## Plugins
+
+The input, filter, and output sections all let you configure plugins. Plugins
+configuration consists of the plugin name followed by a block of settings for
+that plugin. For example, how about two file inputs:
+
+    input {
+      file {
+        path => "/var/log/messages"
+        type => "syslog"
+      }
+
+      file {
+        path => "/var/log/apache/access.log"
+        type => "apache"
+      }
+    }
+
+The above configures a two file separate inputs. Both set two
+configuration settings each: path and type. Each plugin has different
+settings for configuring it, seek the documentation for your plugin to
+learn what settings are available and what they mean. For example, the
+[file input][fileinput] documentation will explain the meanings of the
+path and type settings.
+
+[fileinput]: inputs/file
+
+## Value Types
+
+The documentation for a plugin may say that a configuration field has a
+certain type.  Examples include boolean, string, array, number, hash,
+etc.
+
+### <a name="boolean"></a>Boolean
+
+A boolean must be either `true` or `false`.
+
+Examples:
+
+    debug => true
+
+### <a name="string"></a>String
+
+A string must be a single value.
+
+Example:
+
+    name => "Hello world"
+
+Single, unquoted words are valid as strings, too, but you should use quotes.
+
+### <a name="number"></a>Number
+
+Numbers must be valid numerics (floating point or integer are OK)
+
+Example:
+
+    port => 33
+
+### <a name="array"></a>Array
+
+An array can be a single string value or multiple. If you specify the same
+field multiple times, it appends to the array.
+
+Examples:
+
+    path => [ "/var/log/messages", "/var/log/*.log" ]
+    path => "/data/mysql/mysql.log"
+
+The above makes 'path' a 3-element array including all 3 strings.
+
+### <a name="hash"></a>Hash
+
+A hash is basically the same syntax as Ruby hashes. 
+The key and value are simply pairs, such as:
+
+    match => { "field1" => "value1", "field2" => "value2", ... }
+
+## <a name="fieldreferences"></a>Field References
+
+All events have properties. For example, an apache access log would have things
+like status code, request path, http verb, client ip, etc. Logstash calls these
+properties "fields." 
+
+In many cases, it is useful to be able to refer to a field by name. To do this,
+you can use the logstash field reference syntax.
+
+By way of example, let us suppose we have this event:
+
+    {
+      "agent": "Mozilla/5.0 (compatible; MSIE 9.0)",
+      "ip": "192.168.24.44",
+      "request": "/index.html"
+      "response": {
+        "status": 200,
+        "bytes": 52353
+      },
+      "ua": {
+        "os": "Windows 7"
+      }
+    }
+
+- the syntax to access fields is `[fieldname]`.
+- if you are only referring to a **top-level field**, you can omit the `[]` and
+simply say `fieldname`.
+- in the case of **nested fields**, like the "os" field above, you need
+the full path to that field: `[ua][os]`.
+
+## <a name="sprintf"></a>sprintf format
+
+This syntax is also used in what logstash calls 'sprintf format'. This format
+allows you to refer to field values from within other strings. For example, the
+statsd output has an 'increment' setting, to allow you to keep a count of
+apache logs by status code:
+
+    output {
+      statsd {
+        increment => "apache.%{[response][status]}"
+      }
+    }
+
+You can also do time formatting in this sprintf format. Instead of specifying a field name, use the `+FORMAT` syntax where `FORMAT` is a [time format](http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html). 
+
+For example, if you want to use the file output to write to logs based on the
+hour and the 'type' field:
+
+    output {
+      file {
+        path => "/var/log/%{type}.%{+yyyy.MM.dd.HH}"
+      }
+    }
+
+## <a name="conditionals"></a>Conditionals
+
+Sometimes you only want a filter or output to process an event under
+certain conditions. For that, you'll want to use a conditional!
+
+Conditionals in logstash look and act the same way they do in programming
+languages. You have `if`, `else if` and `else` statements. Conditionals may be
+nested if you need that.
+
+The syntax is follows:
+
+    if EXPRESSION {
+      ...
+    } else if EXPRESSION {
+      ...
+    } else {
+      ...
+    }
+
+What's an expression? Comparison tests, boolean logic, etc!
+
+The following comparison operators  are supported:
+
+* equality, etc: ==  !=  <  >  <=  >= 
+* regexp: =~ !~ 
+* inclusion: in, not in
+
+The following boolean operators are supported:
+
+* and, or, nand, xor
+
+The following unary operators are supported:
+
+* !
+
+Expressions may contain expressions. Expressions may be negated with `!`.
+Expressions may be grouped with parentheses `(...)`. Expressions can be long
+and complex.
+
+For example, if we want to remove the field `secret` if the field
+`action` has a value of `login`:
+
+    filter {
+      if [action] == "login" {
+        mutate { remove => "secret" }
+      }
+    }
+
+The above uses the field reference syntax to get the value of the
+`action` field. It is compared against the text `login` and, when equal,
+allows the mutate filter to do delete the field named `secret`
+
+How about a more complex example?
+
+* alert nagios of any apache events with status 5xx
+* record any 4xx status to elasticsearch
+* record all status code hits via statsd
+
+How about telling nagios of any http event that has a status code of 5xx?
+
+    output {
+      if [type] == "apache" {
+        if [status] =~ /^5\d\d/ {
+          nagios { ...  }
+        } else if [status] =~ /^4\d\d/ {
+          elasticsearch { ... }
+        }
+
+        statsd { increment => "apache.%{status}" }
+      }
+    }
+
+You can also do multiple expressions in a single condition:
+
+    output {
+      # Send production errors to pagerduty
+      if [loglevel] == "ERROR" and [deployment] == "production" {
+        pagerduty {
+          ...
+        }
+      }
+    }
+
+## Further Reading
+
+For more information, see [the plugin docs index](index)