logstash-lib 1.3.2

data/docs/flags.md

@@ -0,0 +1,43 @@
+---
+title: Command-line flags - logstash
+layout: content_right
+---
+# Command-line flags
+
+## Agent
+
+The logstash agent has the following flags (also try using the '--help' flag)
+
+<dl>
+<dt> -f, --config CONFIGFILE </dt>
+<dd> Load the logstash config from a specific file, directory, or a
+wildcard. If given a directory or wildcard, config files will be read
+from the directory in alphabetical order. </dd>
+<dt> -e CONFIGSTRING </dt>
+<dd> Use the given string as the configuration data. Same syntax as the
+config file. If not input is specified, 'stdin { type => stdin }' is
+default. If no output is specified, 'stdout { debug => true }}' is
+default. </dd>
+<dt> -w, --filterworkers COUNT </dt>
+<dd> Run COUNT filter workers (default: 1) </dd>
+<dt> --watchdog-timeout TIMEOUT </dt>
+<dd> Set watchdog timeout value in seconds. Default is 10.</dd>
+<dt> -l, --log FILE </dt>
+<dd> Log to a given path. Default is to log to stdout </dd>
+<dt> -v </dt>
+<dd> Increase verbosity. There are multiple levels of verbosity available with
+'-vv' currently being the highest </dd>
+<dt> --pluginpath PLUGIN_PATH </dt>
+<dd> A colon-delimted path to find other logstash plugins in </dd>
+</dl>
+
+
+## Web
+
+<dl>
+<dt> -a, --address ADDRESS </dt>
+<dd>Address on which to start webserver. Default is 0.0.0.0.</dd>
+<dt> -p, --port PORT</dt>
+<dd>Port on which to start webserver. Default is 9292.</dd>
+</dl>
+

data/docs/generate_index.rb

@@ -0,0 +1,28 @@
+#!/usr/bin/env ruby
+
+require "erb"
+
+if ARGV.size != 1
+  $stderr.puts "No path given to search for plugin docs"
+  $stderr.puts "Usage: #{$0} plugin_doc_dir"
+  exit 1
+end
+
+def plugins(glob)
+  files = Dir.glob(glob)
+  names = files.collect { |f| File.basename(f).gsub(".html", "") }
+  return names.sort
+end # def plugins
+
+basedir = ARGV[0]
+docs = {
+  "inputs" => plugins(File.join(basedir, "inputs/*.html")),
+  "codecs" => plugins(File.join(basedir, "codecs/*.html")),
+  "filters" => plugins(File.join(basedir, "filters/*.html")),
+  "outputs" => plugins(File.join(basedir, "outputs/*.html")),
+}
+
+template_path = File.join(File.dirname(__FILE__), "index.html.erb")
+template = File.new(template_path).read
+erb = ERB.new(template, nil, "-")
+puts erb.result(binding)

data/docs/index.html.erb

@@ -0,0 +1,56 @@
+---
+title: logstash docs index
+layout: content_right
+---
+<div id="doc_index_container">
+
+  <h3> for users </h3>
+  <ul>
+    <li> <a href="https://download.elasticsearch.org/logstash/logstash/logstash-%VERSION%-flatjar.jar"> download logstash %VERSION% </a> </li>
+    <li> <a href="repositories">Repositories</a> </li>
+    <li> <a href="configuration"> configuration file overview </a> </li>
+    <li> <a href="configuration#conditionals">conditionals</a> </li>
+    <li> <a href="configuration#fieldreferences">referring to fields [like][this]</a> </li>
+    <li> <a href="configuration#sprintf">using the %{fieldname} syntax</a> </li>
+
+    <li> <a href="life-of-an-event"> the life of an event in logstash </a> </li>
+    <li> <a href="flags"> command-line flags </a> </li>
+  </ul>
+
+  <h3> for developers </h3>
+  <ul>
+    <li> <a href="extending"> writing your own plugins </a> </li>
+  </ul>
+
+  <h3> use cases and tutorials </h3>
+
+  <ul>
+    <li> <a href="tutorials/getting-started-simple"> getting started (standalone) </a> </li>
+    <li> <a href="tutorials/getting-started-centralized"> getting started (centralized) </a> </li>
+    <li> <a href="tutorials/10-minute-walkthrough"> 10-minute walkthrough</a> - a simple walkthrough to show you how to configure the logstash agent to process events and even old logs. </li>
+    <li> <a href="tutorials/metrics-from-logs"> Gathering metrics from logs </a> - take metrics from logs and ship them to graphite, ganglia, and more. </li>
+    <li> <a href="tutorials/just-enough-rabbitmq-for-logstash">Just enough RabbitMQ for Logstash </a> - Get a quick primer on RabbitMQ and how to use it in Logstash! </li>
+  </ul>
+
+  <h3> books and articles </h3>
+
+  <ul>
+    <li> <a href="http://www.logstashbook.com">The LogStash Book </a> - An introductory LogStash book. </li>
+  </ul>
+
+  <h3> plugin documentation </h3>
+<% docs.each do |type, paths| -%>
+  <div class="doc_index_section">
+    <h3><%= type %></h3>
+    <ul>
+<% paths.each do |path| -%>
+<%   name = File.basename(path).gsub(".html", "") -%>
+      <li>
+      <a href="<%= "#{type}/#{name}" %>"><%= name %></a>
+      </li>
+<% end -%>
+    </ul>
+  </div>
+<% end -%>
+</div>
+<div class="clear"></div>

data/docs/learn.md

@@ -0,0 +1,46 @@
+---
+title: Learn - logstash
+layout: content_right
+---
+# What is logstash?
+
+logstash is a tool for managing your logs.
+
+It helps you take logs and other event data from your systems and move it into
+a central place. logstash is open source and completely free. You can find
+support on the mailing list and on IRC.
+
+For an overview of logstash and why you would use it, you should watch the
+presentation I gave at CarolinaCon 2011: 
+[video here](http://carolinacon.blip.tv/file/5105901/). This presentation covers
+logstash, how you can use it, some alternatives, logging best practices,
+parsing tools, etc. Video also below:
+
+<!--
+<embed src="http://blip.tv/play/gvE9grjcdQI" type="application/x-shockwave-flash" width="480" height="296" allowscriptaccess="always" allowfullscreen="true"></embed>
+
+The slides are available online here: [slides](http://goo.gl/68c62). The slides
+include speaker notes (click 'actions' then 'speaker notes').
+-->
+<iframe width="480" height="296" src="http://www.youtube.com/embed/RuUFnog29M4" frameborder="0" allowfullscreen="allowfullscreen"></iframe>
+
+The slides are available online here: [slides](http://semicomplete.com/presentations/logstash-puppetconf-2012/).
+
+## Getting Help
+
+There's [documentation](.) here on this site. If that isn't sufficient, you can
+email the mailing list (logstash-users@googlegroups.com). Further, there is also
+an IRC channel - #logstash on irc.freenode.org.
+
+If you find a bug or have a feature request, file them
+on <http://logstash.jira.com/>. (Honestly though, if you prefer email or irc
+for such things, that works for me, too.)
+
+## Download It
+
+[Download logstash-%VERSION%](https://download.elasticsearch.org/logstash/logstash/logstash-%VERSION%-flatjar.jar)
+
+## What's next?
+
+Try the [standalone logstash guide](tutorials/getting-started-simple) for a simple
+real-world example getting started using logstash.

data/docs/life-of-an-event.md

@@ -0,0 +1,109 @@
+---
+title: the life of an event - logstash
+layout: content_right
+---
+# the life of an event
+
+The logstash agent is an event pipeline.
+
+## The Pipeline
+
+The logstash agent is a processing pipeline with 3 stages: inputs -> filters ->
+outputs. Inputs generate events, filters modify them, outputs ship them
+elsewhere.
+
+Internal to logstash, events are passed from each phase using internal queues.
+It is implemented with a 'SizedQueue' in Ruby. SizedQueue allows a bounded
+maximum of items in the queue such that any writes to the queue will block if
+the queue is full at maximum capacity.
+
+Logstash sets each queue size to 20. This means only 20 events can be pending
+into the next phase - this helps reduce any data loss and in general avoids
+logstash trying to act as a data storage system. These internal queues are not
+for storing messages long-term.
+
+## Fault Tolerance
+
+Starting at outputs, here's what happens when things break.
+
+An output can fail or have problems because of some downstream cause, such as
+full disk, permissions problems, temporary network failures, or service
+outages. Most outputs should keep retrying to ship any events that were
+involved in the failure.
+
+If an output is failing, the output thread will wait until this output is
+healthy again and able to successfully send the message. Therefore, the output
+queue will stop being read from by this output and will eventually fill up with
+events and block new events from being written to this queue.
+
+A full output queue means filters will block trying to write to the output
+queue. Because filters will be stuck, blocked writing to the output queue, they
+will stop reading from the filter queue which will eventually cause the filter
+queue (input -> filter) to fill up.
+
+A full filter queue will cause inputs to block when writing to the filters.
+This will cause each input to block, causing each input to stop processing new
+data from wherever that input is getting new events.
+
+In ideal circumstances, this will behave similarly to when the tcp window
+closes to 0, no new data is sent because the receiver hasn't finished
+processing the current queue of data, but as soon as the downstream (output)
+problem is resolved, messages will begin flowing again..
+
+## Thread Model
+
+The thread model in logstash is currently:
+
+    input threads | filter worker threads | output worker
+
+Filters are optional, so you will have this model if you have no filters
+defined:
+
+    input threads | output worker
+
+Each input runs in a thread by itself. This allows busier inputs to not be
+blocked by slower ones, etc. It also allows for easier containment of scope
+because each input has a thread.
+
+The filter thread model is a 'worker' model where each worker receives an event
+and applies all filters, in order, before emitting that to the output queue.
+This allows scalability across CPUs because many filters are CPU intensive
+(permitting that we have thread safety). 
+
+The default number of filter workers is 1, but you can increase this number
+with the '-w' flag on the agent.
+
+The output worker model is currently a single thread. Outputs will receive
+events in the order they are defined in the config file. 
+
+Outputs may decide to buffer events temporarily before publishing them,
+possibly in a separate thread. One example of this is the elasticsearch output
+which will buffer events and flush them all at once, in a separate thread. This
+mechanism (buffering many events + writing in a separate thread) can improve
+performance so the logstash pipeline isn't stalled waiting for a response from
+elasticsearch.
+
+## Consequences and Expectations
+
+Small queue sizes mean that logstash simply blocks and stalls safely during
+times of load or other temporary pipeline problems. There are two alternatives
+to this - unlimited queue length and dropping messages. Unlimited queues grow
+grow unbounded and eventually exceed memory causing a crash which loses all of
+those messages. Dropping messages is also an undesirable behavior in most cases.
+
+At a minimum, logstash will have probably 3 threads (2 if you have no filters).
+One input, one filter worker, and one output thread each.
+
+If you see logstash using multiple CPUs, this is likely why. If you want to
+know more about what each thread is doing, you should read this:
+<http://www.semicomplete.com/blog/geekery/debugging-java-performance.html>.
+
+Threads in java have names, and you can use jstack and top to figure out who is
+using what resources. The URL above will help you learn how to do this.
+
+On Linux platforms, logstash will label all the threads it can with something
+descriptive. Inputs will show up as "<inputname" and filter workers as
+"|worker" and outputs as ">outputworker" (or something similar).  Other threads
+may be labeled as well, and are intended to help you identify their purpose
+should you wonder why they are consuming resources!
+

data/docs/logging-tool-comparisons.md

@@ -0,0 +1,60 @@
+---
+title: Logging tools comparisons - logstash
+layout: content_right
+---
+# Logging tools comparison
+
+The information below is provided as "best effort" and is not strictly intended
+as a complete source of truth. If the information below is unclear or incorrect, please
+email the logstash-users list (or send a pull request with the fix) :)
+
+Where feasible, this document will also provide information on how you can use
+logstash with these other projects.
+
+# logstash
+
+Primary goal: Make log/event data and analytics accessible.
+
+Overview: Where your logs come from, how you store them, or what you do with
+them is up to you. Logstash exists to help make such actions easier and faster.
+
+It provides you a simple event pipeline for taking events and logs from any
+input, manipulating them with filters, and sending them to any output. Inputs
+can be files, network, message brokers, etc. Filters are date and string
+parsers, grep-like, etc. Outputs are data stores (elasticsearch, mongodb, etc),
+message systems (rabbitmq, stomp, etc), network (tcp, syslog), etc.
+
+It also provides a web interface for doing search and analytics on your
+logs.
+
+# graylog2
+
+[http://graylog2.org/](http://graylog2.org)
+
+_Overview to be written_
+
+You can use graylog2 with logstash by using the 'gelf' output to send logstash
+events to a graylog2 server. This gives you logstash's excellent input and
+filter features while still being able to use the graylog2 web interface.
+
+# whoops
+
+[whoops site](http://www.whoopsapp.com/)
+
+_Overview to be written_
+
+A logstash output to whoops is coming soon - <https://logstash.jira.com/browse/LOGSTASH-133>
+
+# flume
+
+[flume site](https://github.com/cloudera/flume/wiki)
+
+Flume is primarily a transport system aimed at reliably copying logs from
+application servers to HDFS.
+
+You can use it with logstash by having a syslog sink configured to shoot logs
+at a logstash syslog input.
+
+# scribe
+
+_Overview to be written_

data/docs/plugin-doc.html.erb

@@ -0,0 +1,91 @@
+---
+title: logstash docs for <%= section %>s/<%= name %>
+layout: content_right
+---
+<h2><%= name %></h2>
+<h3>Milestone: <a href="../plugin-milestones"><%= @milestone %></a></h3>
+
+<%= description %>
+
+<% if !@flags.empty? -%>
+<!-- Flags are deprecated 
+<h3> Flags </h3>
+
+This plugin provides the following flags:
+
+<dl>
+<% @flags.each do |flag, description| -%>
+<%# Prefix flag with plugin name. %>
+  <dt> <%= flag.gsub(/^--/, "--#{name}-") %> </dt>
+  <dd> <%= description %> </dd>
+<% end -%>
+</dl>
+
+... flags are deprecated -->
+
+<% end -%>
+
+<h3> Synopsis </h3>
+
+This is what it might look like in your config file:
+
+<pre><code><% if section == "codec" -%>
+# with an input plugin:
+# you can also use this codec with an output.
+input { 
+  file { 
+    codec =&gt; <%= synopsis.split("\n").map { |l| "  #{l}" }.join("\n") %>
+  }
+}
+<% else -%>
+<%= section %> {
+  <%= synopsis %>
+}
+<% end -%></code></pre>
+
+<h3> Details </h3>
+
+<% sorted_attributes.each do |name, config| -%>
+<%
+     if name.is_a?(Regexp)
+       name = "/" + name.to_s.gsub(/^\(\?-mix:/, "").gsub(/\)$/, "") + "/"
+       is_regexp = true
+     else
+       is_regexp = false
+     end
+-%>
+<h4> 
+  <a name="<%= name %>">
+    <%= name %><%= " (required setting)" if config[:required] %>
+    <%= " <strong>DEPRECATED</strong>" if config[:deprecated] %>
+</a>
+</h4>
+
+<ul>
+<% if config[:deprecated] -%>
+  <li> DEPRECATED WARNING: This config item is deprecated. It may be removed in a further version. </li>
+<% end -%>
+<% if is_regexp -%>
+  <li> The configuration attribute name here is anything that matches the above regular expression. </li>
+<% end -%>
+<% if config[:validate].is_a?(Symbol) -%>
+  <li> Value type is <a href="../configuration#<%= config[:validate] %>"><%= config[:validate] %></a> </li>
+<% elsif config[:validate].nil? -%>
+  <li> Value type is <a href="../configuration#string">string</a> </li>
+<% elsif config[:validate].is_a?(Array) -%>
+  <li> Value can be any of: <%= config[:validate].map(&:inspect).join(", ") %> </li>
+<% end -%>
+<% if config.include?(:default) -%>
+  <li> Default value is <%= config[:default].inspect %> </li>
+<% else -%>
+  <li> There is no default value for this setting. </li>
+<% end -%>
+</ul>
+
+<%= config[:description] %>
+
+<% end -%>
+
+<hr>
+
+This is documentation from <a href="https://github.com/logstash/logstash/blob/v<%= LOGSTASH_VERSION %>/<%= file %>"><%= file %></a>

data/docs/plugin-milestones.md

@@ -0,0 +1,41 @@
+---
+title: Plugin Milestones - logstash
+layout: content_right
+---
+# Plugin Milestones
+
+Plugins (inputs/outputs/filters/codecs) have a milestone label in logstash.
+This is to provide an indicator to the end-user as to the kinds of changes
+a given plugin could have between logstash releases.
+
+The desire here is to allow plugin developers to quickly iterate on possible
+new plugins while conveying to the end-user a set of expectations about that
+plugin.
+
+## Milestone 1
+
+Plugins at this milestone need your feedback to improve! Plugins at this
+milestone may change between releases as the community figures out the best way
+for the plugin to behave and be configured.
+
+## Milestone 2
+
+Plugins at this milestone are more likely to have backwards-compatibility to
+previous releases than do Milestone 1 plugins. This milestone also indicates
+a greater level of in-the-wild usage by the community than the previous
+milestone.
+
+## Milestone 3
+
+Plugins at this milestone have strong promises towards backwards-compatibility.
+This is enforced with automated tests to ensure behavior and configuration are
+consistent across releases.
+
+## Milestone 0
+
+This milestone appears at the bottom of the page because it is very
+infrequently used.
+
+This milestone marker is used to generally indicate that a plugin has no
+active code maintainer nor does it have support from the community in terms
+of getting help.