splog 0.0.2

checksums.yaml

@@ -0,0 +1,15 @@
+---
+!binary "U0hBMQ==":
+  metadata.gz: !binary |-
+    MGQ5MTc5ZGUwMGJjNGE0NDBiZWViMWIzYmY3M2JkODZhNDMzMGJiOQ==
+  data.tar.gz: !binary |-
+    NmIxNDZjZDdjYTlkMDU3NmQ2YzQzOGRjYWExNjUyY2IzNDg2YWJhNw==
+!binary "U0hBNTEy":
+  metadata.gz: !binary |-
+    NTk2OTlhOWU0ZTJlNzQxOTkxYTg4NTNmOWZiNTQyNzYyMzY5ODg1MzFjYjFl
+    YzVjMzVlMTE4YTM3N2M5ZGVmZDVkZTQ1YTRjNmYyNjRiZGMzYmYzMDZmMDlk
+    MjMxY2U5OTZlODZkN2VkZmZmYWI1MzY5ZWE4NDk2NDE0NjY0OGI=
+  data.tar.gz: !binary |-
+    ZDk3ZjQxZTA3YWZlOWZkYzc1OWEzNWVhZjAxMzc4Yjg4NzE2NGQ1MGZlNjQz
+    YWMzY2NhMGM0MjBmNzY0Y2M1MDMyZGI0ZGNlMjdhMjlhYTkzMmMyYWIxNjhh
+    ZDdmOGE3OWMzNDhiZjNjYTdjMzBkNjQxMjk5MTkwN2IyMTk1YTk=

data/.gitignore

@@ -0,0 +1,54 @@
+# OS X
+.DS_Store
+.AppleDouble
+.LSOverride
+Icon
+
+
+# Thumbnails
+._*
+#
+# # Files that might appear on external disk
+.Spotlight-V100
+.Trashes
+
+# Jetbrains
+.idea
+
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+
+### Ruby ###
+*.gem
+*.rbc
+.bundle
+.config
+coverage
+InstalledFiles
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+
+# YARD artifacts
+.yardoc
+_yardoc
+doc/

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in splog.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2013 Samuel Mendenhall
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,150 @@
+# Splog -- Simple Log Parsing
+
+Splog allows you to define simple regular expressions in a yaml file that determine how to parse each line of any log file.
+
+There are many similar solutions out in the wild but no solution that was simple and concise and could be configured to work with most any logfile.  Most solutions are locked into a particular format, like Apache logs, or behind a paywall.
+
+Splog solves this by allowing you to define regular expressions to split apart a line, any line, from any file, into a hash and even mapping the regular expression groups to various types like a DateTime or an Integer.
+
+Splog also provides a feature that few other existing solutions do, and that is what I'll call fast forward parsing of a log.  Let's say you have a Java app server log that has exceptions.  Those exceptions should roll up into the originating log line.  That can be easily configured by setting a match forward regular expression.  
+
+By default unmatched lines are rolled into the previously matche line.  This also passively solves the issue of multiline log files which really should be one log entry with multiple lines.
+
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'splog'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install splog
+
+If you want to persist to mongo make sure you have mongo installed: http://www.mongodb.org/downloads
+
+## Usage
+
+#### Getting started
+
+* Install splog
+* Create a ~/.splog.yml -- The quickest approach here is to copy from https://github.com/engineersamuel/splog/blob/master/examples/.splog.yml which has several patterns to get you started.
+* Run splog on a log file!  See the the Example section below.
+
+#### Pretty printing json
+
+There are many libraries out there to pretty print json.  I happen to be partial to https://github.com/rafeca/prettyjson
+
+#### Examples
+
+If you want to test a pattern on a large log file just head that file and pipe it to splog
+    
+    head -n 2 path_to/some_log | splog -p pattern_name -o json | prettyjson
+
+Parsing an Apache access log to stdout by specifying the filename directly.  The default output is stdout so no need to specify that directly
+
+    splog -p apache_common -f path_to/access_log
+
+Same command with piping
+
+    cat path_to/apache_log | splog -p apache_common
+
+And if you just want to test that the pattern is working with the log file and the log file is huge, head the results and pipe them
+
+    head -n 10 examples/access_log | splog -p apache_common
+
+Of course it isn't so easy to tell from stdout if the logs were parsed, I recommend json for that.  This will give you a clear visual that all logs were parsed as you expected them to be parsed
+
+    head -n 10 examples/access_log  | splog -p apache_common -o json | prettyjson
+
+Setting a custom key on each parsed line
+
+    head -n 10 examples/access_log | splog -p apache_common -o json -k server_1 
+
+Disabling the md5 on the hash
+
+    head -n 10 examples/access_log | splog -p apache_common -o json -k server_1 --no-md5
+
+Persisting the log to mongo.  Set -o with no arg so that no output to stdout.  The -d mongo says look in yaml for a `db_refs` key of mongo and read the database infomration to persist to.  Since the default is to hash the parsed log entry to `_id` then repeated executions of the command will result in no duplicated mongo entries.
+
+    head -n 2 examplese/access_log | splog -p apache_common -o -k server_1 -d mongo
+    monogo; use splog; db.logs.find().pretty()
+    {
+        "_id" : "e2304358eb62489fae8beb78d90a39a8",
+        "Host" : "127.0.0.103",
+        "Identity" : "-",
+        "User" : "-",
+        "Time" : ISODate("2013-10-03T16:31:00Z"),
+        "Request" : "STATUS / HTTP/1.0",
+        "Status" : 200,
+        "Size" : 86,
+        "Referer" : "-",
+        "UserAgent" : "ClusterListener/1.0",
+        "key" : "server1"
+    }
+    {
+        "_id" : "1507c52901c51ea2cc7ca687187c82e5",
+        "Host" : "127.0.0.224",
+        "Identity" : "-",
+        "User" : "-",
+        "Time" : ISODate("2013-10-03T16:31:03Z"),
+        "Request" : "STATUS / HTTP/1.0",
+        "Status" : 200,
+        "Size" : 86,
+        "Referer" : "-",
+        "UserAgent" : "ClusterListener/1.0",
+        "key" : "server1"
+    }
+
+
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
+
+## Building Splog in Development
+
+`rake build`
+`rake release` 
+
+#### Examples of executing the bin/splog in development
+
+    ruby -Ilib ./bin/splog
+    ruby -Ilib ./bin/splog -p apache_common -f test/examples/apache/simple_access_log -o json
+    ruby -Ilib ./bin/splog -p apache_common -f test/examples/apache/simple_access_log -o json | prettyjson
+    ruby -Ilib ./bin/splog -p jboss_log4j_common -f test/examples/jboss/multiline_match_server.log -c test/examples/jboss/.splog.yml -o json | prettyjson
+
+#### Executing the spec tests in dev
+
+    bundle exec rspec test
+
+#### A few performance measurements
+
+Note that everything is done in a streaming manner with Ruby Enumerators.  No file is ever read completely.  At most a file is read two lines at a time.  The memory requirements should be minimal due to this.
+
+Parsing a 1m JBoss log to json:
+
+    time ruby -Ilib ./bin/splog -p jboss_log4j_common -f ./server.log -o json  
+    2.99s user 0.04s system 88% cpu 3.437 total
+
+Parsing a 1m JBoss log to with no writing to stdout:
+
+    time ruby -Ilib ./bin/splog -p jboss_log4j_common -f ./server.log -o 
+    2.81s user 0.02s system 99% cpu 2.834 total
+
+Parsing a 45m Apache access log:
+
+    time ruby -Ilib ./bin/splog -p apache_common -f ./access_log -o
+    44.49s user 0.10s system 99% cpu 44.596 total
+
+## Dependencies
+
+Splog is compatible with Ruby 1.9+

data/Rakefile

@@ -0,0 +1 @@
+require "bundler/gem_tasks"

data/bin/splog

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require 'splog'
+parser = Splog::LogParser.new
+parser.cli(ARGV)

data/examples/.splog.yml

@@ -0,0 +1,96 @@
+db_refs:
+    mongo:
+        type: 'mongo'
+        host: "127.0.0.1"
+        port: "27017"
+        user: ""
+        pass: ""
+        db: "splog"
+        collection: "logs"
+
+# Data Types supported ['DateTime' 'Integer', 'Float', 'String']
+apache_common:
+    db_key:
+        delim: '_'
+        # Define an array to create the compound key to define this log file
+        # This is most useful when combined saving the log to mongo
+        # You can specify any command line option here to be dereferenced
+        # In this case if I passed in -k prod -f access_log the key would be
+        # "prod_access_log"
+        values:
+            - '--key'
+            - '--file'
+        # If the filename is sufficiently complex like access_log_prod_1020PMEST
+        # I'd recommend something like below with -k prod_access_log to force the key
+        # to be easier to facilitate mongo searches
+        #values:
+        #    - '--key'
+        
+    # Delimeter to join the regex array lines.  Apache access is space delimited
+    delim: '\s*'
+    # Regex to be joined to parse each log line
+    # ex. 127.0.0.1 - - [25/Sep/2013:15:35:27 -0400] "INFO / HTTP/1.0" 200 - "-" "ClusterListener/1.0"
+    regex:
+        - '(?<Host>\S+)'                   # host %h
+        - '(?<Identity>\S+)'               # indent %l (unused)
+        - '(?<User>\S+)'                   # user %u
+        - '\[(?<Time>.+)\]'                # time %t
+        - '"(?<Request>.+)"'               # request "%r"
+        - '(?<Status>[0-9]+)'              # status %>s
+        - '(?<Size>\S+)'                   # size %b (careful can be '-')
+        - '"(?<Referer>.*)"'               # referer "%{Referer}i"
+        - '"(?<UserAgent>.*)"'             # user agent "%{User-agent}i"
+    # Define which name in the hash to append contents to based on the regex group
+    #append_key_name: ''
+    # Any mappings not specified are assumed to be of type String
+    mapping:
+        - name: Time
+          data_type: DateTime
+          format: '%d/%b/%Y:%H:%M:%S %z'
+        - name: Status 
+          data_type: Integer
+        - name: Size
+          data_type: Integer
+
+# Data Types supported ['DateTime' 'Integer', 'Float', 'String']
+jboss_log4j_common:
+    db_key:
+        delim: '_'
+        values:
+            - '--key'
+            - '--file'
+    # Delimeter to join the regex array lines.  Apache access is space delimited
+    delim: '\s+'
+    # Regex to be joined to parse each log line
+    # <!-- The default pattern: Date Priority [Category] (Thread) Message\n -->
+    # <param name="ConversionPattern" value="%d %-5p [%c] (%t) %m%n"/>
+    # (?<Date>.*?)\s+(?<Priority>WARN|ERROR|INFO|TRACE|DEBUG)\s+\[(?<Category>.*?)\]\s+\((?<Thread>.*?)\)\s+(?<Message>.*)
+    regex:
+        - '(?<Date>.*?)'
+        - '(?<Priority>WARN|ERROR|INFO|TRACE|DEBUG)'
+        - '\[(?<Category>.*?)\]'
+        - '\((?<Thread>.*?)\)'
+        - '(?<Message>.*)'
+    # For lines that don't match the regex above, define a key name if you want the log line
+    # TO be appended to a previously matched line
+    unmatched_append_key_name: 'Message'
+    # Define a regex pattern to define any matched lines to append to the previously matched line
+    # Example: 03 Oct 2013 20:16:55,309 ERROR [stderr] (MSC service thread 1-3)  at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:116)
+    match_forward_regex:
+        - '(?<Date>.*?)'
+        - '(?<Priority>WARN|ERROR|INFO|TRACE|DEBUG)'
+        - '\[(?<Category>.*?)\]'
+        - '\((?<Thread>.*?)\)'
+        - '(?<Message>\tat\s.*?)'
+
+    # Define the hash key to read from, otherwise default to the entire line
+    match_forward_keyname_source: 'Message'
+
+    # Define the hash key to read to, Mandatory if match_forward_regex defined
+    match_forward_keyname_dest: 'Message'
+    # Any mappings not specified are assumed to be of type String
+    #mapping:
+    #    - name: Status 
+    #      data_type: Integer
+    #    - name: Size
+    #      data_type: Integer

data/examples/access_log

@@ -0,0 +1,50 @@
+127.0.0.103 - - [03/Oct/2013:12:31:00 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.224 - - [03/Oct/2013:12:31:03 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.229 - - [03/Oct/2013:12:32:36 -0400] "STATUS / HTTP/1.0" 500 648 "-" "ClusterListener/1.0"
+127.0.0.229 - - [03/Oct/2013:12:32:46 -0400] "INFO / HTTP/1.0" 200 3403 "-" "ClusterListener/1.0"
+127.0.0.229 - - [03/Oct/2013:12:32:46 -0400] "CONFIG / HTTP/1.0" 200 - "-" "ClusterListener/1.0"
+127.0.0.229 - - [03/Oct/2013:12:32:46 -0400] "ENABLE-APP / HTTP/1.0" 200 - "-" "ClusterListener/1.0"
+127.0.0.228 - - [03/Oct/2013:12:33:23 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.120 - - [03/Oct/2013:12:33:30 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.226 - - [03/Oct/2013:12:33:59 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.126 - - [03/Oct/2013:12:34:00 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.224 - - [03/Oct/2013:12:34:26 -0400] "REMOVE-APP / HTTP/1.0" 200 - "-" "ClusterListener/1.0"
+127.0.0.225 - - [03/Oct/2013:12:34:50 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.224 - - [03/Oct/2013:12:35:01 -0400] "INFO / HTTP/1.0" 200 3652 "-" "ClusterListener/1.0"
+127.0.0.100 - - [03/Oct/2013:12:35:28 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.122 - - [03/Oct/2013:12:35:40 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.102 - - [03/Oct/2013:12:36:49 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.103 - - [03/Oct/2013:12:37:28 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.229 - - [03/Oct/2013:12:39:04 -0400] "STATUS / HTTP/1.0" 500 648 "-" "ClusterListener/1.0"
+127.0.0.229 - - [03/Oct/2013:12:39:14 -0400] "INFO / HTTP/1.0" 200 3317 "-" "ClusterListener/1.0"
+127.0.0.229 - - [03/Oct/2013:12:39:14 -0400] "CONFIG / HTTP/1.0" 200 - "-" "ClusterListener/1.0"
+127.0.0.229 - - [03/Oct/2013:12:39:14 -0400] "ENABLE-APP / HTTP/1.0" 200 - "-" "ClusterListener/1.0"
+127.0.0.1 - - [03/Oct/2013:12:39:39 -0400] "GET /x/ HTTP/1.1" 503 435 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.3; MS-RTC LM 8;)"
+127.0.0.1 - - [03/Oct/2013:12:39:49 -0400] "GET /x/ HTTP/1.1" 503 435 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.3; MS-RTC LM 8;)"
+127.0.0.228 - - [03/Oct/2013:12:39:51 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.120 - - [03/Oct/2013:12:39:58 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.226 - - [03/Oct/2013:12:40:27 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.126 - - [03/Oct/2013:12:40:28 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.225 - - [03/Oct/2013:12:41:18 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.224 - - [03/Oct/2013:12:41:19 -0400] "CONFIG / HTTP/1.0" 200 - "-" "ClusterListener/1.0"
+127.0.0.224 - - [03/Oct/2013:12:41:19 -0400] "ENABLE-APP / HTTP/1.0" 200 - "-" "ClusterListener/1.0"
+127.0.0.100 - - [03/Oct/2013:12:41:56 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.122 - - [03/Oct/2013:12:42:08 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.224 - - [03/Oct/2013:12:42:42 -0400] "REMOVE-APP / HTTP/1.0" 200 - "-" "ClusterListener/1.0"
+127.0.0.102 - - [03/Oct/2013:12:43:17 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.224 - - [03/Oct/2013:12:43:23 -0400] "INFO / HTTP/1.0" 200 3652 "-" "ClusterListener/1.0"
+127.0.0.103 - - [03/Oct/2013:12:43:56 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.229 - - [03/Oct/2013:12:45:32 -0400] "STATUS / HTTP/1.0" 500 648 "-" "ClusterListener/1.0"
+127.0.0.229 - - [03/Oct/2013:12:45:42 -0400] "INFO / HTTP/1.0" 200 3317 "-" "ClusterListener/1.0"
+127.0.0.229 - - [03/Oct/2013:12:45:42 -0400] "CONFIG / HTTP/1.0" 200 - "-" "ClusterListener/1.0"
+127.0.0.229 - - [03/Oct/2013:12:45:42 -0400] "ENABLE-APP / HTTP/1.0" 200 - "-" "ClusterListener/1.0"
+127.0.0.228 - - [03/Oct/2013:12:46:19 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.120 - - [03/Oct/2013:12:46:26 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.226 - - [03/Oct/2013:12:46:55 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.126 - - [03/Oct/2013:12:46:56 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.225 - - [03/Oct/2013:12:47:46 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.100 - - [03/Oct/2013:12:48:24 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.122 - - [03/Oct/2013:12:48:36 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"
+127.0.0.224 - - [03/Oct/2013:12:49:41 -0400] "CONFIG / HTTP/1.0" 200 - "-" "ClusterListener/1.0"
+127.0.0.224 - - [03/Oct/2013:12:49:41 -0400] "ENABLE-APP / HTTP/1.0" 200 - "-" "ClusterListener/1.0"
+127.0.0.102 - - [03/Oct/2013:12:49:45 -0400] "STATUS / HTTP/1.0" 200 86 "-" "ClusterListener/1.0"

data/lib/splog/version.rb

@@ -0,0 +1,3 @@
+module Splog
+  VERSION = '0.0.2'
+end