rubylogparser 0.1.0

data/EXAMPLES.txt

@@ -0,0 +1,137 @@
+= RubyLogParser Examples
+
+See the examples directory for the full source of the scripts presented below.
+
+Also, the "Log Parser Toolkit" (more info in GUIDE.txt) has many examples of
+how to use Log Parser.
+
+== File System
+
+Log Parser has access to the file system including the various file attributes.
+In this example, the top 10 largest files are found in the Windows directory
+(this *should* work for systems with either C:\winnt or C:\windows but the first
+directory that starts with c:\win will be used) and returned in descending order.
+
+Note that in the paramters for the open_query method, the input is the file
+system ('FS'), the query returns the results to standard output ('INTO STDOUT')
+in comma separated value ('CSV') format.
+
+The data is read into a hash and then formatted for output to the screen. Once
+all the data is returned, the various query statistics (Processed, Output and
+Time) are also printed.
+
+  require 'rubylogparser.rb'
+  
+  lp = RubyLogParser.new
+  lp.open_query('FS', 'Select Top 10 name, size INTO STDOUT from c:/win*.* ORDER BY size DESC', 'CSV', nil)
+
+  while hash = lp.read_hash do
+    p "#{hash['Name'].ljust(60)} #{hash['Size'].rjust(12)}\n"
+  end
+
+  p "Processed: " + (lp.elements_processed.nil? ? "0" : "#{lp.elements_processed}") + "\n"
+  p "Output: " + (lp.elements_output.nil? ? "0" : "#{lp.elements_output}") + "\n"
+  p "Time: " + (lp.execution_time.nil? ? "0" : "#{lp.execution_time}") + " seconds\n"
+
+== Event Log
+
+Log Parser can be very helpful in scanning the event log. This example scans the
+System event log and counts the number of entries for each source. This makes it
+easy to see what events are occuring most frequently.
+
+Note that in the paramters for the open_query method, the input is the event
+log ('EVT') and the actual query specifies the System log (other options
+include security, application, IE), the query returns the results to standard
+output ('INTO STDOUT') in comma separated value ('CSV') format.
+
+In this example, each line of data is read into an array for further processing
+(printed to the screen) and the query statistics printed. 
+
+  require 'rubylogparser.rb'
+
+  lp = RubyLogParser.new
+
+  sql = "
+    Select
+      SourceName, 
+      count(*) AS number_of_events
+    INTO STDOUT  
+    FROM System
+    GROUP BY SourceName
+    ORDER BY number_of_events desc
+  "
+  
+  lp.open_query('EVT', sql, 'CSV', nil)
+  while array = lp.read_array_line do
+    p "#{array[0].ljust(35)} #{array[1].rjust(8)}\n"
+  end
+
+  p "Processed: " + (lp.elements_processed.nil? ? "0" : "#{lp.elements_processed}") + "\n"
+  p "Output: " + (lp.elements_output.nil? ? "0" : "#{lp.elements_output}") + "\n"
+  p "Time: " + (lp.execution_time.nil? ? "0" : "#{lp.execution_time}") + " seconds\n"
+
+== Registry
+
+Log Parser also can scan the registry. 
+
+Note that in the paramters for the open_query method, the input is the registry
+('REG') and the actual query specifies the HKLM\Software keys, the query returns
+the results to standard output ('INTO STDOUT') in comma separated value ('CSV')
+format.
+
+And again, like the Event Log example, the output is read line by line into an
+array for processing (formatted and printed to the screen) and then the query
+statistics printed. However, errors are commonly encountered due to Windows
+locking registry keys from being read and these keys are displayed before
+printing the query statistics.
+
+  require 'rubylogparser.rb'
+
+  lp = RubyLogParser.new
+
+  sql = "
+    Select  Path,
+            ValueName
+    INTO    STDOUT
+    FROM    HKLM\\SOFTWARE
+    WHERE   LastWriteTime >= SUB(SYSTEM_TIMESTAMP(), TIMESTAMP('0000-01-02', 'yyyy-MM-dd'))
+  "
+
+  lp.open_query('REG', sql, 'CSV', {'e' => 100})
+
+  while array = lp.read_array_line do
+    p "#{array[0].ljust(80)} #{array[1].rjust(25)}\n"
+  end
+
+  p "Parse errors:\n" + lp.parse_errors.to_s + "\n\n"
+  p "Statistics:\n"
+  p "Processed: " + (lp.elements_processed.nil? ? "0" : "#{lp.elements_processed}") + "\n"
+  p "Output: " + (lp.elements_output.nil? ? "0" : "#{lp.elements_output}") + "\n"
+  p "Time: " + (lp.execution_time.nil? ? "0" : "#{lp.execution_time}") + " seconds\n"
+
+== The queryInfo switch
+
+The queryInfo switch is useful to see how Log Parser will process the various
+input options including the data types the fields will be returned as.
+Truthfully, this switch is probably most useful when used directly from the
+command line.
+
+An additional merit of this example is to show how additional command line
+options can be passed to the open_query method. In this case, the queryInfo
+parameter is set to true which overrides the default false value. Another example
+that would ignore warnings ('iw') and choose the queryInfo would be passed to the
+open_query method as {'queryInfo' => true, 'iw' => true}.
+
+  require 'rubylogparser.rb'
+  lp = RubyLogParser.new
+
+  sql = "
+    Select Top 3 timegenerated AS EventTime, message AS Message
+    FROM system
+    WHERE eventid='6005'
+    ORDER BY timegenerated DESC
+  "
+
+  lp.open_query('EVT', sql, nil, {'queryInfo' => true})
+
+  puts "#{lp.formatted_queryInfo}"

data/GUIDE.txt

@@ -0,0 +1,97 @@
+= Getting Started With LogParser
+This guide is meant to get you started using the rubylogparser gem to interface
+with Microsoft's Log Parser.
+
+A book that I highly recommend is the "Log Parser Toolkit" by Gabriele Giuseppini
+and Mark Burnett (ISBN 1-932266-52-6). It is a treasure chest of examples for
+analyzing log files, event logs, investigating intrusions, security auditing,
+formatting, reporting, charting and so much more. It also includes a nice
+reference for input and output formats including commonly used parameters. In
+short, you'll get a lot more out of Log Parser if you look at how people have
+been using it to do some really interesting things.
+
+The included examples really just scratches the surface of what is possible, but
+should be enough information to get you really going!
+
+== Log Parser overview
+The basic architecture of Log Parser is that it allows an input source to be
+queried by an SQL statement and the results sent to a variety of output formats.
+
+Input sources may be log files, event logs, the file system, the registry, XML
+files, ETW trace logs, among others. The output formats include CSV files, log
+file formats, Syslogs, XML, charts and more.
+
+A basic use of Log Parser is to choose an input source, select and/or filter it
+with an SQL statement and output it to a desired format. Hence, conversion and
+filtering of data is greatly simplified.
+
+== Functionality of rubylogparser gem
+The goal of the rubylogparser gem is to scrape the Log Parser output into Ruby
+arrays or hashes for further processing. For instance, the event log can be
+monitored and specific events pushed to a database via DBI, email alerts sent
+using NET/SMTP, Ruport for reporting, etc.
+
+To best use this gem and populate the Ruby data structures properly, the output
+from Log Parser needs to be in CSV format sent to standard output (STDOUT).
+Internally, this gem opens Log Parser using an IO.popen call and output from
+Log Parser is subsequently read from a pipe using readline commands.  That it
+why the output needs to be specified as STDOUT so the data can be read from the
+pipe. The CSV line is then split into the respective fields and returned as an
+array or hash with the key values being the field/column names.
+
+If Log Parser warnings and query statistics are enabled (they are by default),
+several class variables (@elements_processed, @elements_output, @execution_time)
+will be populated after all the data has been read in. Any errors will be added
+to an array of strings called @parse_errors.
+
+== Let's Fetch Some Information!
+First thing is first.  Make sure that you've required rubylogparser and that you
+instantiate a new logparser object:
+
+ require 'rubygems'
+ require 'rubylogparser'
+
+ lp = RubyLogParser.new
+ 
+Now we'll create a SQL statement to count the number of distinct events in the
+System event log and send the output 'INTO STDOUT'.
+
+  # This SQL statement orders System events by order of their SourceName
+  sql = "
+    Select
+      SourceName, 
+      count(*) AS number_of_events
+    INTO STDOUT  
+    FROM System
+    GROUP BY SourceName
+    ORDER BY number_of_events desc
+  "
+
+Here we create the Log Parser process and specify:
+ - input format ('EVT' for event log)
+ - the SQL statement stored in the sql variable
+ - the output format ('CSV' for comma separated values)
+
+If we wanted to override any of the default log parser parameters, we could
+replace the nil in the method call with a hash containing the command line
+parameter values such as {'iw' => true} to ignore warnings. Run the Log Parser
+executable from the command line such as "logparser.exe -h" to see the various
+command line switches and additional help.
+
+  lp.open_query('EVT', sql, 'CSV', nil)
+
+Next we read the Log Parser output line by line into an array and format the
+output to be printed to the screen. See the examples/files.rb example for using
+a hash to retrieve output.
+
+  while array = lp.read_array_line do
+    p "#{array[0].ljust(35)} #{array[1].rjust(8)}\n"
+  end
+
+Once all output has been processed, the query statistics and any warnings/errors
+are available.
+
+  p "Parse errors:\n" + lp.parse_errors.to_s + "\n\n"
+  p "Processed: " + (lp.elements_processed.nil? ? "0" : "#{lp.elements_processed}") + "\n"
+  p "Output: " + (lp.elements_output.nil? ? "0" : "#{lp.elements_output}") + "\n"
+  p "Time: " + (lp.execution_time.nil? ? "0" : "#{lp.execution_time}") + " seconds\n"

data/History.txt

@@ -0,0 +1,5 @@
+== 0.1.0 / 2008-01-10
+
+* Initial Release (Pilchuck)
+  * Birthday!
+

data/Manifest.txt

@@ -0,0 +1,12 @@
+EXAMPLES.txt
+GUIDE.txt
+History.txt
+Manifest.txt
+README.txt
+Rakefile
+examples/event_log.rb
+examples/files.rb
+examples/queryinfo.rb
+examples/registry.rb
+lib/rubylogparser.rb
+test/test_rubylogparser.rb

data/README.txt

@@ -0,0 +1,107 @@
+rubylogparser
+    by Jim Clark (jimclark@ieee.org)
+    http://rubyforge.org/projects/rubylogparser/
+
+== DESCRIPTION:
+  
+The RubyLogParser library is used for interacting with Microsoft's Log Parser 
+tool (http://www.microsoft.com/downloads/results.aspx?freetext=Log%20Parser).
+
+Log Parser is a powerful, versatile tool that provides universal query
+access to text-based data such as log files, XML files and CSV files, as
+well as key data sources on the Windows operating system such as the
+Event Log, the Registry, the file system, and Active Directory.
+
+The most important benefit of using this library is that the Log Parser
+output can be returned in arrays or hashes for further processing in Ruby.
+
+== FEATURES/PROBLEMS:
+  
+The rubylogparser gem allows complete control over the Log Parser executable.
+Although charts, XML files, log file formats and other output formats are
+possible, the main goal is to scrape the Log Parser output into Ruby data
+structures for further processing. Currently, arrays and hashes are supported.
+
+See the following synopsis for a quick code sample or for a more detailed
+explanation, check out the GUIDE[link://files/GUIDE_txt.html].  
+Also, check out the EXAMPLES[link://files/EXAMPLES_txt.html] file.
+
+== SYNOPSIS:
+
+The following code will search the c:\win*.* directory (matching c:\windows or
+c:\winnt on most systems) for the 10 largest files. It make take a couple of
+minutes to run. The input for Log Parser is the file system ('FS') and output
+is directed to standard output (STDOUT) in comma separated value ('CSV') format
+which is where the rubylogparser gem scrapes the Log Parser output from. Data
+is returned in a hash with the hash keys corresponding to the selected fields in
+the SQL statement.
+
+  require 'rubylogparser.rb'
+  lp = RubyLogParser.new
+  lp.open_query('FS', 'Select Top 10 name, size INTO STDOUT from c:/win*.* ORDER BY size DESC', 'CSV', nil)
+  while hash = lp.read_hash do
+    p " #{hash['Name'].ljust(60)} #{hash['Size'].rjust(12)}\n"
+  end
+
+== REQUIREMENTS:
+
+Testing was done in the following environment:
+
+* ruby 1.8.6
+
+* Microsoft Log Parser 2.2
+
+Other software versions may work but are completely untested. Testing with Rails
+and Ruby 1.9 will be done in early 2008.
+
+== INSTALL:
+
+To use this library, first make sure that you have Microsoft's Log Parser tool
+installed in your system. If you choose not to use the default install location,
+make sure to add the directory where the logparser.exe executable is installed
+in on the path.
+
+Next, install the gem using "gem install -r rubylogparser"
+
+RubyLogParser will automatically search for the Log Parser executable in these
+locations:
+
+* c:\Program Files\IIS Resources\Log Parser 2.2\LogParser.exe (Default install location for the IIS Resource Kit)
+
+* c:\Program Files\Log Parser 2.2\LogParser.exe (Default location when downloaded and installed without the IIS Resource kit)
+
+* LogParser.exe on the path somewhere
+
+If the LogParser.exe executable cannot be found, the "test_valid_executable"
+test will fail. If not installed in a default location, adding the installation
+directory to the path is the easiest way to allow the executable to be found.
+
+== ACKNOWLEDGEMENTS:
+
+Thank you to Ryan Davis for guidance and troubleshooting and making this 
+library better. 
+
+== LICENSE:
+
+(The MIT License)
+
+Copyright (c) 2008 Jim Clark (jimclark@ieee.org)
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,21 @@
+require 'rubygems'
+require 'hoe'
+$:.unshift(File.dirname(__FILE__) + "/lib")
+require 'rubylogparser'
+
+Hoe.new('rubylogparser', RubyLogParser::VERSION) do |p|
+  p.rubyforge_name = 'rubylogparser'
+  p.author = 'Jim Clark'
+  p.email = 'jimclark@ieee.org'
+  p.summary = "RubyLogParser provides a wrapper around Microsoft's Log Parser executable."
+  p.description = "RubyLogParser enables the output from Microsoft's Log Parser to be processed by Ruby data structures (arrays and hashes)."
+  p.remote_rdoc_dir = '' # Release to root  
+  p.url = 'http://rubyforge.org/projects/rubylogparser/'
+  p.changes = p.paragraphs_of('History.txt', 0..1).join("\n\n")
+  p.need_tar = false
+  p.need_zip = true
+end
+
+desc "Release and publish documentation"
+task :repubdoc => [:release, :publish_docs]
+

data/examples/event_log.rb

@@ -0,0 +1,30 @@
+#!c:/ruby/bin/ruby.exe
+
+$:.unshift File.join(File.dirname(__FILE__), "..", "lib")
+require 'rubylogparser.rb'
+
+lp = RubyLogParser.new
+
+# This SQL statement orders System events by order of their SourceName
+sql = "
+Select
+  SourceName, 
+  count(*) AS number_of_events
+INTO STDOUT  
+FROM System
+GROUP BY SourceName
+ORDER BY number_of_events desc
+"
+
+lp.open_query('EVT', sql, 'CSV', nil)
+
+i=1
+while array = lp.read_array_line do
+  p "#{i}.".ljust(4) + " #{array[0].ljust(35)} #{array[1].rjust(8)}\n"
+  i += 1
+end
+
+p "Parse errors:\n" + lp.parse_errors.to_s + "\n\n"
+p "\nProcessed: " + (lp.elements_processed.nil? ? "0" : "#{lp.elements_processed}") + "\n"
+p "Output: " + (lp.elements_output.nil? ? "0" : "#{lp.elements_output}") + "\n"
+p "Time: " + (lp.execution_time.nil? ? "0" : "#{lp.execution_time}") + " seconds\n"

data/examples/files.rb

@@ -0,0 +1,19 @@
+#!c:/ruby/bin/ruby.exe
+
+$:.unshift File.join(File.dirname(__FILE__), "..", "lib")
+require 'rubylogparser.rb'
+
+lp = RubyLogParser.new
+
+# Note: This query may take several minutes to run
+lp.open_query('FS', 'Select Top 10 name, size INTO STDOUT from c:/win*.* ORDER BY size DESC', 'CSV', nil)
+
+i=1
+while hash = lp.read_hash do
+  p "#{i}.".ljust(4) + " #{hash['Name'].ljust(60)} #{hash['Size'].rjust(12)}\n"
+  i += 1
+end
+
+p "Processed: " + (lp.elements_processed.nil? ? "0" : "#{lp.elements_processed}") + "\n"
+p "Output: " + (lp.elements_output.nil? ? "0" : "#{lp.elements_output}") + "\n"
+p "Time: " + (lp.execution_time.nil? ? "0" : "#{lp.execution_time}") + " seconds\n"

data/examples/queryinfo.rb

@@ -0,0 +1,18 @@
+#!c:/ruby/bin/ruby.exe
+
+$:.unshift File.join(File.dirname(__FILE__), "..", "lib")
+require 'rubylogparser'
+
+# This SQL selects a common event from the system log. Since the query will be
+# evaluated but not actually executed, it doesn't really matter if the specific
+# event will be found in the event log. 
+sql = "
+Select Top 3 timegenerated AS EventTime, message AS Message
+FROM system
+WHERE eventid='6005'
+ORDER BY timegenerated DESC
+"
+
+lp = RubyLogParser.new
+lp.open_query('EVT', sql, nil, {'queryInfo' => true})
+puts "#{lp.formatted_queryInfo}"

data/examples/registry.rb

@@ -0,0 +1,30 @@
+#!c:/ruby/bin/ruby.exe
+
+$:.unshift File.join(File.dirname(__FILE__), "..", "lib")
+require 'rubylogparser.rb'
+
+lp = RubyLogParser.new
+
+# This SQL statement filters registry keys from HKLM\Software that were written
+# to in the last day
+sql = "
+Select  Path,
+        ValueName
+INTO    STDOUT
+FROM    HKLM\\SOFTWARE
+WHERE   LastWriteTime >= SUB(SYSTEM_TIMESTAMP(), TIMESTAMP('0000-01-02', 'yyyy-MM-dd'))
+"
+
+lp.open_query('REG', sql, 'CSV', {'e' => 100})
+
+i=1
+while array = lp.read_array_line do
+  p "#{i}.".ljust(5) + " #{array[0].ljust(80)} #{array[1].rjust(25)}\n"
+  i += 1  
+end
+
+p "Parse errors:\n" + lp.parse_errors.to_s + "\n\n"
+p "Statistics:\n"
+p "Processed: " + (lp.elements_processed.nil? ? "0" : "#{lp.elements_processed}") + "\n"
+p "Output: " + (lp.elements_output.nil? ? "0" : "#{lp.elements_output}") + "\n"
+p "Time: " + (lp.execution_time.nil? ? "0" : "#{lp.execution_time}") + " seconds\n"

data/lib/rubylogparser.rb

@@ -0,0 +1,330 @@
+# rubylogparser.rb : Ruby wrapper around Microsoft's LogParser executable.
+# Methods and variables for interacting with the LogParser process.  Most of
+# these methods are for retrieving data using hashes or arrays to allow 
+# the maximum flexibility in retrieving the LogParser output.
+#
+# Author: James A. Clark (jimclark at ieee dot org)
+#
+# Copyright (c) 2007, All Rights Reserved.
+#
+# This is free software.  You may modify and redistribute this freely under
+# your choice of the GNU General Public License or the Ruby License. 
+#
+# See LICENSE and COPYING for details
+#
+
+require 'csv'
+
+class RubyLogParser
+  VERSION = "0.1.0"
+  
+  # Stores the various options that will be used to build the command line that
+  # the process is eventually started with
+  attr_accessor :cmd_options
+  
+  # The command line string that IO.popen uses to open the LogParser parser
+  attr_accessor :cmd_string
+  
+  # The pipe to the IO.popen process to read LogParser output data
+  attr_accessor :logparser_pipe
+
+  # Column headings on the output data are controlled with the " -q" quiet mode
+  # switch. When enabled (the default), the various column names are stored in
+  # this array. This is populated immediately after the process is created and
+  # before any data is read in using the self.read_hash or self.read_array
+  # methods.
+  attr_reader :field_names
+  
+  # LogParser reports parse errors when it can't do something it expects to.
+  # One example is when access is denied to registry keys. All parse errors are
+  # stuffed into this array so they can be examined at the end of processing.
+  attr_reader :parse_errors
+  
+  # Output when using the " -queryInfo" switch will be stored here
+  attr_reader :queryInfo
+  
+  # Query statistics are turned on by default and controlled with the " -stats"
+  # switch. The output includes Elements Processed, Elements Output, and
+  # Execution time in seconds and is available in these class variables after
+  # all input has been read.
+  attr_reader :elements_processed, :elements_output, :execution_time
+  
+  # Stores the location of the LogParser.exe file. 
+  attr_writer :logparser_exe
+
+  # Initialize is used to check that the Log Parser executable can be found
+  # and to config the various command line defaults.
+  def initialize(lp_executable = nil)
+    if self.valid_executable(lp_executable) == false
+      raise "The LogParser executable file cannot be found."  
+    end
+    self.set_defaults
+  end 
+
+  # Simple test to make sure the LogParser executable can be found. By default, it
+  # checks the IIS Resource Kit install path, the stand-alone LogParser install path,
+  # and then if the LogParser.exe file is found on the path.
+  # If an executable path is provided, this is tried instead allowing the user to
+  # pick the desired path if one or more LogParser versions are installed.
+  def valid_executable(lp_executable = nil)
+    found = false
+    if lp_executable.nil? then
+      file_locations = ['c:\Program Files\IIS Resources\Log Parser 2.2\LogParser.exe', 'c:\Program Files\Log Parser 2.2\LogParser.exe', 'LogParser.exe']
+    else
+      file_locations = [ lp_executable ]
+    end
+    
+    file_locations.each {|location|
+      if File::executable? location then
+        found = true
+        @logparser_exe = location
+      end
+    }
+    return found
+  end
+
+  # LogParser has a variety of switches to control the input, output, SQL and
+  # process options. Most can be viewed by opening up LogParser at a command
+  # prompt and typing "c:\Logparser>  logparser -h". There are additional
+  # switches that are not documented in the "logparser -h" output used to for
+  # controlling things like chart appearance. See the self.open_query method
+  # for more information.
+  def set_defaults
+    @cmd_options = {
+      'sql' => nil,  # SQL query empty to start with
+      'i' => nil,    # input mode
+      'o' => 'CSV',  # output mode
+      'q' => false,  # quiet mode - when true field headings supressed
+      'e' => -1,     # max number of errors 
+      'iw' => false, # ignore warnings
+      'stats' => true, # display statistics after executing query
+      'c' => false,     # use built-in conversion query
+      'multiSite' => false, # send BIN conversion output to multiple files - see help
+      'saveDefaults' => false, # save specificed options as default values
+      'restoreDefaults' => false, # restore factory defaults
+      'rtp' => 0,   # -1 turns off "Press a key..." prompts normally found in DOS window when NAT output selected
+      'queryInfo' => false # display query processing info
+    }
+    return @cmd_options
+  end
+  
+  # Input formats are checked against valid formats to catch simple errors
+  def valid_input_format(input)
+    if input.nil? then
+      return true
+    end
+    input_formats = %w(IISW3C NCSA IIS IISODBC BIN IISMSID HTTPERR URLSCAN
+                       CSV TSV W3C XML EVT ETW NETMON REG ADS TEXTLINE
+                       TEXTWORD FS COM)
+    input_formats.each {|format| return true if format==input.upcase}
+    return false
+  end
+
+  # Output formats are checked against valid formats to catch simple errors
+  def valid_output_format(output)
+    if output.nil? then
+      return true
+    end
+    output_formats = %w(CSV TSV XML DATAGRID CHART SYSLOG NEUROVIEW NAT
+                        W3C IIS SQL TPL NULL)
+    output_formats.each{|format| return true if format==output.upcase}
+    return false
+  end
+  
+  # The LogParser process is opened using a string formatted for a command line.
+  # All of the command line options can be viewed in more detail by running
+  # "logparser -h" to see online help. The command line string is built using
+  # the currently configured options including any values in the input hash
+  # used to override or add specific functionality.
+  def build_command_string(options = nil)
+    @cmd_string = nil
+    if !options.nil? then
+      options.each {|key, value| @cmd_options[key] = value}
+    end
+    if (valid_input_format(@cmd_options['i']) &&
+       valid_output_format(@cmd_options['o']) &&
+       @cmd_options['sql'] != nil) then
+      @cmd_string = "\"#{@logparser_exe}\"" 
+      @cmd_string += " -i:#{@cmd_options['i']}" unless @cmd_options['i'].nil?
+      @cmd_options['sql'].gsub!(/[\r\n]/, ' ') # remove any linefeeds from formatted SQL statement
+      @cmd_string += " \"#{@cmd_options['sql']}\""
+      @cmd_string += " -o:#{@cmd_options['o']}" unless @cmd_options['o'].nil?
+      @cmd_string += " -q:ON" if cmd_options['q']
+      @cmd_string += " -e:#{@cmd_options['e']}" if @cmd_options['e'] >= 0
+      @cmd_string += " -iw:ON" if @cmd_options['iw']
+      @cmd_string += " -stats:OFF" if @cmd_options['stats'] == false
+      @cmd_string += " -c" if @cmd_options['c']
+      @cmd_string += " -multiSite:ON" if @cmd_options['multiSite']
+      @cmd_string += " -saveDefaults" if @cmd_options['saveDefaults']
+      @cmd_string += " -restoreDefaults" if @cmd_options['restoreDefaults']
+      @cmd_string += " -rtp:#{@cmd_options['rtp']}" if @cmd_options['rtp'] != 0 # turns off "Press a key"
+      @cmd_string += " -queryInfo" if @cmd_options['queryInfo']
+    end
+    return @cmd_string
+  end
+
+  # LogParser needs a data input source, a SQL selection query, and an output
+  # location. A hash can also be passed to override default values or provide
+  # additional command line switches. An interesting exception that needs to be
+  # handled is LogParser's "-queryInfo" switch which makes LogParser evaluate
+  # the input parameters and return info on the various column datatypes.
+  def open_query(input, query, output, options)
+    @cmd_options['i'] = input
+    @cmd_options['sql'] = query
+    @cmd_options['o'] = output
+    @cmd_string = self.build_command_string(options)
+
+    self.create_process
+    
+    if @cmd_options['queryInfo'] == true then
+      @queryInfo = []
+      @logparser_pipe.each do |line|
+        @queryInfo << line
+      end
+    elsif @cmd_options['q'] == false then
+      self.field_definitions
+    end
+  end
+
+  # The LogParser process is created using IO.popen and then subsequent output
+  # from LogParser is read via a pipe.
+  def create_process
+    begin
+      @logparser_pipe = IO.popen(@cmd_string, "rb")
+    rescue Exception => e
+      puts "Exit status is: #{e.status}"
+    end
+  end
+
+  # Reads a single line of output from the LogParser pipe until an EOF is
+  # detected. LogParser will first output all of the query data and then a "\r\n"
+  # blank line followed by any errors or query statistics. Hence, once this
+  # method detects the blank line, all remaining LogParser output is expected
+  # to be error or statistic related. 
+  def readline
+    begin
+      line = @logparser_pipe.readline
+      if line == "\r\n" then
+        line_errors_and_stats(line)
+      else
+        line.chomp!
+        return line
+      end
+    rescue EOFError
+      return nil
+    end
+  end
+  
+  # The first line of output contains the field names (by default). This
+  # information is used to name the various hash keys when data is returned in
+  # a hash. 
+  def field_definitions
+    @field_names = Array.new
+    names_line = self.readline
+    if !names_line.nil? then
+      @field_names = names_line.split(/,/)
+    end
+  end
+
+  # The output from LogParser is return in comma separated value (CSV) format.
+  # Using split to determine the fields is much faster than CSV. Occasionally
+  # however, a field will have a comma in it which will cause split to break
+  # a field in two. In cases where the actual number of fields returned by split
+  # does not match the expected number, the CSV library is called to try the
+  # split. This usually fixes the problem.
+  def read_array_line
+    next_line = self.readline
+    if !next_line.nil? then 
+      array_line = next_line.split(/,/)
+      if array_line.length == @field_names.length then
+        return array_line
+      else
+        CSV.parse(next_line) {|row|
+          if row.length == @field_names.length then
+            return row
+          else
+            line_errors_and_stats(next_line)
+            return nil
+          end
+        }
+      end
+    end
+    return nil
+  end
+
+  # Once LogParser finishes outputting query data, it then outputs any errors
+  # followed by query statistics (unless specifically turned off). Errors are
+  # pushed into the @parse_errors array and statistics are added to the
+  # appropriate class variables.
+  def line_errors_and_stats(line_input)
+    begin
+      while 1 > 0
+        case line_input
+        when /Task completed with parse errors/
+          @parse_errors = Array.new
+          until ((line_input = @logparser_pipe.readline) == "\r\n")
+            @parse_errors.push(line_input)
+          end
+        when /Statistics:/
+          while line = @logparser_pipe.readline
+            case line
+            when /Elements processed:\s*([\d,]+)\n*/i
+              @elements_processed = $1
+            when /Elements output:\s*([\d,]+)\n*/i
+              @elements_output = $1
+            when /Execution time:\s*([\d\.]+) seconds\n*/i
+              @execution_time = $1
+            end
+          end
+        end
+        line_input = @logparser_pipe.readline
+      end
+    rescue EOFError
+      return nil
+    end
+  end
+  
+  # To return the data in a hash, the data line is first read into an array.
+  # Then the field keys (initially loaded by the field_definitions method when
+  # the LogParser process was created), are matched up with the respective data
+  # values.
+  def read_hash
+    data_hash = Hash.new
+    data_array = read_array_line
+    if !data_array.nil? then 
+      @field_names.each_index {|x|
+        data_hash[@field_names[x]] = data_array[x]
+      }
+      return data_hash
+    else
+      return nil
+    end
+  end
+  
+  # When data is best processed all at once, this method will return an array
+  # of arrays. The array will have a header row with the field names by default
+  # unless LogParser is started in "quiet" mode (i.e. -q:ON).
+  def read_array(blnHeader_row)
+    data_array = Array.new
+    i=0
+    if blnHeader_row
+      data_array[i] = @field_names
+      i += 1
+    end
+    while array = self.read_array_line do
+      data_array[i] = array
+      i += 1
+    end
+    return data_array
+  end
+  
+  # A simple convenience method when running LogParser using the -queryInfo
+  # switch. The resulting output is combined into a single string for easier
+  # printing.
+  def formatted_queryInfo
+    temp = String.new
+    @queryInfo.each{|line| temp << line}
+    return temp
+  end
+  
+end

data/test/test_rubylogparser.rb

@@ -0,0 +1,43 @@
+#!/usr/bin/env ruby -w
+
+$:.unshift File.join(File.dirname(__FILE__), "..", "lib")
+require 'test/unit'
+require 'rubylogparser.rb'
+
+class Test_RubyLogParser < Test::Unit::TestCase
+  def setup
+    @logparser = RubyLogParser.new
+  end
+  
+  def test_valid_executable
+    assert_equal true, @logparser.valid_executable(nil)
+    assert_equal false, @logparser.valid_executable('c:\someotherfilename.exe')
+  end
+  
+  def test_set_defaults
+    test_defaults = @logparser.set_defaults
+    assert_equal 'CSV', test_defaults['o']
+    assert_equal false, test_defaults['queryInfo']
+  end
+  
+  def test_valid_input_type
+    assert_equal true, @logparser.valid_input_format("CSV")
+    assert_equal true, @logparser.valid_input_format("csv")
+    assert_equal false, @logparser.valid_input_format("xxx")    
+  end
+
+  def test_valid_output_type
+    assert_equal true, @logparser.valid_output_format("CSV")
+    assert_equal true, @logparser.valid_output_format("csv")
+    assert_equal false, @logparser.valid_output_format("xxx")    
+  end
+  
+  def test_build_command_string
+    @logparser.cmd_options['i'] = "EVT"
+    @logparser.cmd_options['sql'] = "SQL Test"
+    @logparser.cmd_options['o'] = "CSV"
+    @logparser.logparser_exe = 'LogParser.exe'
+    assert_equal '"LogParser.exe" -i:EVT "SQL Test" -o:CSV -queryInfo', @logparser.build_command_string({'queryInfo' => true})
+  end
+
+end

metadata

@@ -0,0 +1,70 @@
+--- !ruby/object:Gem::Specification 
+rubygems_version: 0.9.4
+specification_version: 1
+name: rubylogparser
+version: !ruby/object:Gem::Version 
+  version: 0.1.0
+date: 2008-01-11 00:00:00 -08:00
+summary: RubyLogParser provides a wrapper around Microsoft's Log Parser executable.
+require_paths: 
+- lib
+email: jimclark@ieee.org
+homepage: http://rubyforge.org/projects/rubylogparser/
+rubyforge_project: rubylogparser
+description: RubyLogParser enables the output from Microsoft's Log Parser to be processed by Ruby data structures (arrays and hashes).
+autorequire: 
+default_executable: 
+bindir: bin
+has_rdoc: true
+required_ruby_version: !ruby/object:Gem::Version::Requirement 
+  requirements: 
+  - - ">"
+    - !ruby/object:Gem::Version 
+      version: 0.0.0
+  version: 
+platform: ruby
+signing_key: 
+cert_chain: 
+post_install_message: 
+authors: 
+- Jim Clark
+files: 
+- EXAMPLES.txt
+- GUIDE.txt
+- History.txt
+- Manifest.txt
+- README.txt
+- Rakefile
+- examples/event_log.rb
+- examples/files.rb
+- examples/queryinfo.rb
+- examples/registry.rb
+- lib/rubylogparser.rb
+- test/test_rubylogparser.rb
+test_files: 
+- test/test_rubylogparser.rb
+rdoc_options: 
+- --main
+- README.txt
+extra_rdoc_files: 
+- EXAMPLES.txt
+- GUIDE.txt
+- History.txt
+- Manifest.txt
+- README.txt
+executables: []
+
+extensions: []
+
+requirements: []
+
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: hoe
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Version::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 1.4.0
+    version: