logstash-lib 1.3.2

data/docs/tutorials/getting-started-simple.md

@@ -0,0 +1,200 @@
+---
+title: Getting Started (Standalone server) - logstash
+layout: content_right
+---
+# Getting started with logstash (standalone server example)
+
+This guide shows how to get you going quickly with logstash on a single,
+standalone server. We'll begin by showing you how to read events from standard
+input (your keyboard) and emit them to standard output. After that, we'll start
+collecting actual log files.
+
+By standalone, I mean that everything happens on a single server: log collection, indexing, and the web interface.
+
+logstash can be run on multiple servers (collect from many servers to a single
+indexer) if you want, but this example shows simply a standalone configuration.
+
+Steps detailed in this guide:
+
+* Download and run logstash
+
+## Problems?
+
+If you have problems, feel free to email the users list
+(logstash-users@googlegroups.com) or join IRC (#logstash on irc.freenode.org)
+
+## logstash
+
+You should download the logstash jar file - if you haven't yet,
+[download it
+now](https://download.elasticsearch.org/logstash/logstash/logstash-%VERSION%-flatjar.jar).
+This package includes most of the dependencies for logstash in it and
+helps you get started quicker.
+
+The configuration of any logstash agent consists of specifying inputs, filters,
+and outputs. For this example, we will not configure any filters.
+
+The inputs are your log files. The output will be elasticsearch. The config
+format should be simple to read and write. The bottom of this document includes
+links for further reading (config, etc) if you want to learn more.
+
+Here is a simple Logstash configuration:
+
+    input { stdin { } }
+    output { stdout { codec => rubydebug } }
+
+Save this to a file called `logstash-simple.conf` and run it like so:
+
+    java -jar logstash-%VERSION%-flatjar.jar agent -f logstash-simple.conf
+
+After a few seconds, type something in the console where you started logstash.
+Maybe `hello`.  You should get some output like so:
+
+    {
+      "message"    => "hello",
+      "@timestamp" => "2013-09-04T00:24:21.707Z",
+      "@version"   => "1",
+      "host"       => "pork"
+    }
+
+If everything is okay, let's move on to a more complex version:
+
+### Saving to Elasticsearch
+
+The recommended storage engine for Logstash is Elasticsearch. If you're running
+Logstash from the jar file or via jruby, you can use an embedded version of
+Elasticsearch for storage.
+
+Using our configuration above, let's change it to look like so:
+
+    input { stdin { type => example } }
+    output { 
+      stdout { codec => rubydebug }
+      elasticsearch { embedded => true }
+    }
+
+We're going to KEEP the existing configuration but add a second output -
+embedded Elasticsearch.  Restart your Logstash (CTRL-C and rerun the java
+command). Depending on the horsepower of your machine, this could take some
+time.  Logstash needs to extract the jar contents to a working directory AND
+start an instance of Elasticsearch.
+
+Let's do our test again by simply typing `test`. You should get the same output to the console.
+Now let's verify that Logstash stored the message in Elasticsearch:
+
+    curl -s http://127.0.0.1:9200/_status?pretty=true | grep logstash
+
+_This assumes you have the `curl` command installed._
+
+You should get back some output like so:
+
+    "logstash-2012.07.02" : {
+      "index" : "logstash-2012.07.02"
+
+This means Logstash created a new index based on today's date. Likely your data is in there as well:
+
+`curl -gs -XGET "http://localhost:9200/logstash-*/_search?pretty&q=type:example"`
+
+This will return a rather large JSON output. We're only concerned with a subset:
+
+      "_index" : "logstash-2013.09.07",
+      "_type" : "logs",
+      "_id" : "iARTN3MtQ-Kaf_x0fZaFwQ",
+      "_score" : 1.4054651, "_source" : {
+        "message": "fizzle",
+        "@timestamp": "2013-09-07T00:42:23.453Z",
+        "@version": "1",
+        "type": "example",
+        "host": "pork"
+      }
+
+Your output may look a little different.
+The reason we're going about it this way is to make absolutely sure that we have all the bits working before adding more complexity.
+
+If you are unable to get these steps working, you likely have something interfering with multicast traffic. This has been known to happen when connected to VPNs for instance.
+For best results, test on a Linux VM or system with less complicated networking. If in doubt, rerun the command with the options `-vv` and paste the output to Github Gist or Pastie.
+Hop on the logstash IRC channel or mailing list and ask for help with that output as reference.
+
+Obviously this is fairly useless this way. Let's add the final step and test with the builtin logstash web ui:
+
+### Testing the webui
+
+We've already proven that events can make it into Elasticsearch. However using
+curl for everything is less than ideal.
+
+Logstash ships with a built-in web interface (called Kibana). Let's restart our
+logstash process with an additional option:
+
+    java -jar logstash-%VERSION%-flatjar.jar agent -f logstash-simple.conf -- web
+
+One important thing to note is that the `web` option is actually its own set of
+commmand-line options. We're essentially starting two programs in one.  This is
+worth remembering as you move to an external Elasticsearch server. The options
+you specify in your logstash.conf have no bearing on the web ui. It has its own
+options.
+
+Again, the reason for testing without the web interface is to ensure that the
+logstash agent itself is getting events into Elasticsearch. This is different
+than the Logstash web ui being able to read them.  As before, we'll need to
+wait a bit for everything to spin up. You can verify that everything is running
+(assuming you aren't running with any `-v` options) by checking the output of
+`netstat`:
+
+    netstat -napt | grep -i LISTEN
+
+What's interesting is that you should see the following ports in use:
+
+- 9200
+- 9300
+- 9301
+- 9302
+- 9292
+
+The `9200` and `9300` ports are the embedded ES listening. The `9301` and `9302` ports are the agent and web interfaces talking to ES. `9292` is the port the web ui listens on.
+
+If you open a browser to http://localhost:9292/ and click on the link in the body, you should see results. If not, switch back to your console, type some test and hit return.
+Refresh the browser page and you should have results!
+
+### Continuing on
+At this point you have a working self-contained Logstash instance. However typing things into stdin is likely not to be what you want.
+
+Here is a sample config you can start with. It defines some basic inputs
+grouped by type and two outputs.
+
+    input {
+      stdin {
+        type => "stdin-type"
+      }
+
+      file {
+        type => "syslog"
+
+        # Wildcards work, here :)
+        path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog" ]
+      }
+    }
+
+    output {
+      stdout { }
+      elasticsearch { embedded => true }
+    }
+
+Put this in a file called "logstash-complex.conf"
+
+Now run it all (again. Be sure to stop your previous Logstash tests!):
+
+    java -jar logstash-%VERSION%-flatjar.jar agent -f logstash-complex.conf -- web
+
+Point your browser at <http://yourserver:9292> and start searching!
+
+*Note*: If things are not working, such as you get an error message while
+searching, like 'SERVICE_UNAVAILABLE' or some other elasticsearch error, you
+should check that your firewall (local, too) is not blocking multicast.
+
+## Further reading
+
+Want to know more about the configuration language? Check out the
+[configuration](../configuration) documentation.
+
+You may have logs on many servers you want to centralize through logstash. To
+learn how to do that, [read this](getting-started-centralized)

data/docs/tutorials/just-enough-rabbitmq-for-logstash.md

@@ -0,0 +1,201 @@
+---
+title: Just Enough RabbitMQ - logstash
+layout: content_right
+---
+
+While configuring your RabbitMQ broker is out of scope for logstash, it's important
+to understand how logstash uses RabbitMQ. To do that, we need to understand a
+little about AMQP.
+
+You should also consider reading
+[this](http://www.rabbitmq.com/tutorials/amqp-concepts.html) at the RabbitMQ
+website.
+
+# Exchanges, queues and bindings; OH MY!
+
+You can get a long way by understanding a few key terms.
+
+## Exchanges
+
+Exchanges are for message **producers**. In Logstash, we map these to
+**outputs**.  Logstash puts messages on exchanges.  There are many types of
+exchanges and they are discussed below.
+
+## Queues
+
+Queues are for message **consumers**. In Logstash, we map these to inputs.
+Logstash reads messages from queues.  Optionally, queues can consume only a
+subset of messages. This is done with "routing keys".
+
+## Bindings
+
+Just having a producer and a consumer is not enough. We must `bind` a queue to
+an exchange.  When we bind a queue to an exchange, we can optionally provide a
+routing key.  Routing keys are discussed below.
+
+## Broker
+
+A broker is simply the AMQP server software. There are several brokers, but this
+tutorial will cover the most common (and arguably popular), [RabbitMQ](http://www.rabbitmq.com).
+
+# Routing Keys
+
+Simply put, routing keys are somewhat like tags for messages. In practice, they
+are hierarchical in nature with the each level separated by a dot:
+
+- `messages.servers.production`
+- `sports.atlanta.baseball`
+- `company.myorg.mydepartment`
+
+Routing keys are really handy with a tool like logstash where you
+can programatically define the routing key for a given event using the metadata that logstash provides:
+
+- `logs.servers.production.host1`
+- `logs.servers.development.host1.syslog`
+- `logs.servers.application_foo.critical`
+
+From a consumer/queue perspective, routing keys also support two types wildcards - `#` and `*`.
+
+- `*` (asterisk) matches any single word.
+- `#` (hash) matches any number of words and behaves like a traditional wildcard.
+
+Using the above examples, if you wanted to bind to an exchange and see messages
+for just production, you would use the routing key `logs.servers.production.*`.
+If you wanted to see messages for host1, regardless of environment you could
+use `logs.servers.%.host1.#`.
+
+Wildcards can be a bit confusing but a good general rule to follow is to use
+`*` in places where you need wildcards for a known element.  Use `#` when you
+need to match any remaining placeholders. Note that wildcards in routing keys
+only make sense on the consumer/queue binding, not in the publishing/exchange
+side.
+
+We'll get into some of that neat stuff below. For now, it's enough to
+understand the general idea behind routing keys.
+
+# Exchange types
+
+There are three primary types of exchanges that you'll see.
+
+## Direct
+
+A direct exchange is one that is probably most familiar to people. Message
+comes in and, assuming there is a queue bound, the message is picked up.  You
+can have multiple queues bound to the same direct exchange. The best way to
+understand this pattern is pool of workers (queues) that read from a direct
+exchange to get units of work. Only one consumer will see a given message in a
+direct exchange.
+
+You can set routing keys on messages published to a direct exchange. This
+allows you do have workers that do different tasks read from the same global
+pool of messages yet consume only the ones they know how to handle.
+
+The RabbitMQ concepts guide (linked below) does a good job of describing this
+visually
+[here](http://www.rabbitmq.com/img/tutorials/intro/exchange-direct.png)
+
+## Fanout
+
+Fanouts are another type of exchange. Unlike direct exchanges, every queue
+bound to a fanout exchange will see the same messages.  This is best described
+as a PUB/SUB pattern. This is helpful when you need broadcast messages to
+multiple interested parties.
+
+Fanout exchanges do NOT support routing keys. All bound queues see all
+messages.
+
+## Topic
+
+Topic exchanges are special type of fanout exchange. Fanout exchanges don't
+support routing keys. Topic exchanges do support them.  Just like a fanout
+exchange, all bound queues see all messages with the additional filter of the
+routing key.
+
+# RabbitMQ in logstash
+
+As stated earlier, in Logstash, Outputs publish to Exchanges. Inputs read from
+Queues that are bound to Exchanges.  Logstash uses the `bunny` RabbitMQ library for
+interaction with a broker. Logstash endeavors to expose as much of the
+configuration for both exchanges and queues.  There are many different tunables
+that you might be concerned with setting - including things like message
+durability or persistence of declared queues/exchanges.  See the relevant input
+and output documentation for RabbitMQ for a full list of tunables.
+
+# Sample configurations, tips, tricks and gotchas
+
+There are several examples in the logstash source directory of RabbitMQ usage,
+however a few general rules might help eliminate any issues.
+
+## Check your bindings
+
+If logstash is publishing the messages and logstash is consuming the messages,
+the `exchange` value for the input should match the `name` in the output.
+
+sender agent
+
+    input { stdin { type = "test" } }
+    output {
+      rabbitmq {
+        exchange => "test_exchange"
+        host => "my_rabbitmq_server"
+        exchange_type => "fanout"
+      }
+    }
+
+receiver agent
+
+    input {
+      rabbitmq {
+        queue => "test_queue"
+        host => "my_rabbitmq_server"
+        exchange => "test_exchange" # This matches the exchange declared above
+      }
+    }
+    output { stdout { debug => true }}
+
+## Message persistence
+
+By default, logstash will attempt to ensure that you don't lose any messages.
+This is reflected in the RabbitMQ default settings as well.  However there are
+cases where you might not want this. A good example is where RabbitMQ is not your
+primary method of shipping.
+
+In the following example, we use RabbitMQ as a sniffing interface. Our primary
+destination is the embedded ElasticSearch instance. We have a secondary RabbitMQ
+output that we use for duplicating messages. However we disable persistence and
+durability on this interface so that messages don't pile up waiting for
+delivery. We only use RabbitMQ when we want to watch messages in realtime.
+Additionally, we're going to leverage routing keys so that we can optionally
+filter incoming messages to subsets of hosts. The exercise of getting messages
+to this logstash agent are left up to the user.
+
+    input { 
+      # some input definition here
+    }
+
+    output {
+      elasticsearch { embedded => true }
+      rabbitmq {
+        exchange => "logtail"
+        host => "my_rabbitmq_server"
+        exchange_type => "topic" # We use topic here to enable pub/sub with routing keys
+        key => "logs.%{host}"
+        durable => false # If rabbitmq restarts, the exchange disappears.
+        auto_delete => true # If logstash disconnects, the exchange goes away
+        persistent => false # Messages are not persisted to disk
+      }
+    }
+
+Now if you want to stream logs in realtime, you can use the programming
+language of your choice to bind a queue to the `logtail` exchange.  If you do
+not specify a routing key, you will see every message that comes in to
+logstash. However, you can specify a routing key like `logs.apache1` and see
+only messages from host `apache1`.
+
+Note that any logstash variable is valid in the key definition. This allows you
+to create really complex routing key hierarchies for advanced filtering.
+
+Note that RabbitMQ has specific rules about durability and persistence matching
+on both the queue and exchange. You should read the RabbitMQ documentation to
+make sure you don't crash your RabbitMQ server with messages awaiting someone
+to pick them up.

data/docs/tutorials/metrics-from-logs.md

@@ -0,0 +1,84 @@
+---
+title: Metrics from Logs - logstash
+layout: content_right
+---
+# Pull metrics from logs
+
+Logs are more than just text. How many customers signed up today? How many HTTP
+errors happened this week? When was your last puppet run?
+
+Apache logs give you the http response code and bytes sent - that's useful in a
+graph. Metrics occur in logs so frequently there are piles of tools available to
+help process them.
+
+Logstash can help (and even replace some tools you might already be using).
+
+## Example: Replacing Etsy's Logster
+
+[Etsy](https://github.com/etsy) has some excellent open source tools. One of
+them, [logster](https://github.com/etsy/logster), is meant to help you pull
+metrics from logs and ship them to [graphite](http://graphite.wikidot.com/) so
+you can make pretty graphs of those metrics.
+
+One sample logster parser is one that pulls http response codes out of your
+apache logs: [SampleLogster.py](https://github.com/etsy/logster/blob/master/logster/parsers/SampleLogster.py)
+
+The above code is roughly 50 lines of python and only solves one specific
+problem in only apache logs: count http response codes by major number (1xx,
+2xx, 3xx, etc). To be completely fair, you could shrink the code required for
+a Logster parser, but size is not strictly the point, here.
+
+## Keep it simple
+
+Logstash can do more than the above, simpler, and without much coding skill:
+
+    input {
+      file { 
+        path => "/var/log/apache/access.log" 
+        type => "apache-access"
+      }
+    }
+
+    filter {
+      grok { 
+        type => "apache-access"
+        pattern => "%{COMBINEDAPACHELOG}" 
+      }
+    }
+
+    output {
+      statsd { 
+        # Count one hit every event by response
+        increment => "apache.response.%{response}" 
+      }
+    }
+
+The above uses grok to parse fields out of apache logs and using the statsd
+output to increment counters based on the response code. Of course, now that we
+are parsing apache logs fully, we can trivially add additional metrics:
+
+    output {
+      statsd {
+        # Count one hit every event by response
+        increment => "apache.response.%{response}"
+
+        # Use the 'bytes' field from the apache log as the count value.
+        count => [ "apache.bytes", "%{bytes}" ]
+      }
+    }
+
+Now adding additional metrics is just one more line in your logstash config
+file. BTW, the 'statsd' output writes to another Etsy tool,
+[statsd](https://github.com/etsy/statsd), which helps build counters/latency
+data and ship it to graphite for graphing.
+
+Using the logstash config above and a bunch of apache access requests, you might end up
+with a graph that looks like this:
+
+![apache response codes graphed with graphite, fed data with logstash](media/frontend-response-codes.png)
+
+The point made above is not "logstash is better than Logster" - the point is
+that logstash is a general-purpose log management and pipelining tool and that
+while you can centralize logs with logstash, you can read, modify, and write
+them to and from just about anywhere.
+