log_sense 1.5.2 → 1.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5f37b2247014af9bccfb8bdd205b54eb51cbef35495904444765648a96e0b9ac
-  data.tar.gz: a9cd03afb6770bf854791b8ec93e87d09532828d894b28f5dde670f1af1b1b1d
+  metadata.gz: 7d269dedfbb6ec6eae3a77491cc5ec7ca6241f388f2658964541cdd3983b8298
+  data.tar.gz: 6f24d23c8d06430b3605aad90522e08818cd18fd6c71c5fe90823cdc9483c81e
 SHA512:
-  metadata.gz: 62981216e38b92e1c3ea227726dccd592c441de32519a4df6f39bac127f55da32e82b4a1a14897b09b7032c7b3f9506a14e80e49dc43bfe8c9d86bfaa340da82
-  data.tar.gz: e5e6180008a12561d688668cffb2ef3d885d7733cf877aafaed0397b9fa0e91dd12a2998b19006dad4fe6b202bd203a00757f86d5d37efee777dbc7be43e5ab1
+  metadata.gz: a63f715b281101a6f61029da3e2bcf5db4d47a537af562507b0184d6c6755eed436afed729c31cc95255bf064cbe9f487917c96d78199bbee25e6d4189469951
+  data.tar.gz: 77c62a24c3c81067dd0288ad606b732e0f112d0d1710b326be9e894aa72a93db4cb0a188c93b54ada728c2eeb0cf7378fb7033e13982c9a0932414c8455d2703

data/CHANGELOG.org

@@ -2,6 +2,33 @@
 #+AUTHOR: Adolfo Villafiorita
 #+STARTUP: showall
 
+* 1.6.0
+
+- [User] New output format =ufw= generates directives to blacklist IPs
+  requesting URLs matching a pattern. For users of the Uncomplicated
+  Firewall.
+- [User] new option =--no-geo= skips geolocation, which is terribly
+  costly in the current implementation.
+- [User] Updated DB-IP country file to Dec 2022 version.
+- [User] Changed name of SQLite output format to sqlite3
+- [User] It is now possible to start analysis from a sqlite3 DB
+  generated by log_sense, breaking parsing and generation in two
+  steps.
+- [User] Check for correctness of I/O formats before launching
+  analysis
+- [User] Streak report has been renames Session.  Limited the number
+  of URLs shown in each session, to avoid buffer?/memory overflows
+  when an IP requests a massive amount of URLs.
+- [User] Added an IP-per-hour visits report.
+- [Code] A rather extensive refactoring of the source code to 
+  remove code duplications and improve code structure. 
+- [Code] Rubocop-ped various files
+- [Code] Added text renderer to DataTable, which sanitizes input and
+  further reduces risks of XSS and log poisoning attacks
+- [Code] CDN links have been ported into the Emitter module and used
+  in the Embedded Ruby Templates (erbs).  This simplifies version
+  updates of Javascript libraries used in reports.
+
 * 1.5.2
 
 - [User] Updated DB-IP country file.

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    log_sense (1.5.2)
+    log_sense (1.5.3)
       browser
       ipaddr
       iso_country_codes
@@ -16,18 +16,20 @@ GEM
       irb (>= 1.3.6)
       reline (>= 0.3.1)
     io-console (0.5.11)
-    ipaddr (1.2.4)
+    ipaddr (1.2.5)
     irb (1.4.1)
       reline (>= 0.3.0)
     iso_country_codes (0.7.8)
+    mini_portile2 (2.8.0)
     minitest (5.15.0)
     rake (12.3.3)
     reline (0.3.1)
       io-console (~> 0.5)
-    sqlite3 (1.4.4)
+    sqlite3 (1.5.4)
+      mini_portile2 (~> 2.8.0)
     terminal-table (3.0.2)
       unicode-display_width (>= 1.1.1, < 3)
-    unicode-display_width (2.2.0)
+    unicode-display_width (2.3.0)
 
 PLATFORMS
   ruby

data/README.org

@@ -9,7 +9,7 @@ Rails logs.  Written in Ruby, it runs from the command line, it is
 fast, and it can be installed on any system with a relatively recent
 version of Ruby.  We tested on Ruby 2.6.9, Ruby 3.0.x and later.
 
-LogSense reports the following data:
+When generating reports, LogSense reports the following data:
 
 - Visitors, hits, unique visitors, bandwidth used
 - Most accessed HTML pages
@@ -22,18 +22,49 @@ LogSense reports the following data:
 - IP Country location, thanks to the DP-IP lite country DB
 - Streaks: resources accessed by a given IP over time
 - Performance of Rails requests
+
+A special output format =ufw= generates rules for the [[https://launchpad.net/ufw][Uncomplicated
+Firewall]] to blacklist IPs requesting URLs matching a specific pattern.
  
 Filters from the command line allow to analyze specific periods and
 distinguish traffic generated by self polls and crawlers.
 
-LogSense generates HTML, txt, and SQLite outputs.
+LogSense generates HTML, txt, ufw, and SQLite outputs.
 
-And, of course, the compulsory screenshot:
+** Apache Report Structure
 
 #+ATTR_HTML: :width 80%
-[[file:./apache-screenshot.png]]
+[[file:./screenshots/apache-screenshot.png]]
+
+
+** Rails Report Structure
+
+#+ATTR_HTML: :width 80%
+[[file:./screenshots/rails-screenshot.png]]
+
+
+** UFW Report
 
+The output format =ufw= generates directives for Uncomplicated
+Firewall blacklisting IPs requesting URLs matching a given pattern.
 
+We use it to blacklist IPs requesting WordPress login pages on our
+websites... since we don't use WordPress for our websites.
+
+*Example*
+
+#+begin_src 
+$ log_sense -f apache -t ufw -i apache.log
+# /users/sign_in/xmlrpc.php?rsd
+ufw deny from 20.212.3.206
+
+# /wp-login.php /wordpress/wp-login.php /blog/wp-login.php /wp/wp-login.php
+ufw deny from 185.255.134.18
+
+...
+#+end_src
+
+   
 * An important word of warning
 
 [[https://owasp.org/www-community/attacks/Log_Injection][Log poisoning]] is a technique whereby attackers send requests with invalidated
@@ -48,9 +79,10 @@ opened or code executed.
 * Motivation
 
 LogSense moves along the lines of tools such as [[https://goaccess.io/][GoAccess]] (which
-strongly inspired the development of Log Sense) and [[https://umami.is/][Umami]], focusing on
-*privacy* and *data-ownership*: the data generated by LogSense is
-stored on your computer and owned by you (like it should be)[fn:1].
+strongly inspired the development of Log Sense) and [[https://umami.is/][Umami]], both
+focusing on *privacy* and *data-ownership*: the data generated by
+LogSense is stored on your computer and owned by you (like it should
+be)[fn:1].
 
 LogSense is also inspired by *static websites generators*: statistics
 are generated from the command line and accessed as static HTML files.
@@ -76,33 +108,30 @@ generated files are then made available on a private area on the web.
   #+RESULTS:
   #+begin_example
   Usage: log_sense [options] [logfile ...]
-          --title=TITLE                Title to use in the report
-      -f, --input-format=FORMAT        Input format (either rails or apache)
-      -i, --input-files=file,file,     Input files (can also be passed directly)
-      -t, --output-format=FORMAT       Output format: html, org, txt, sqlite. See below for available formats
-      -o, --output-file=OUTPUT_FILE    Output file
-      -b, --begin=DATE                 Consider entries after or on DATE
-      -e, --end=DATE                   Consider entries before or on DATE
-      -l, --limit=N                    Limit to the N most requested resources (defaults to 100)
-      -w, --width=WIDTH                Maximum width of long columns in textual reports
-      -r, --rows=ROWS                  Maximum number of rows for columns with multiple entries in textual reports
-      -c, --crawlers=POLICY            Decide what to do with crawlers (applies to Apache Logs)
-      -n, --no-selfpolls               Ignore self poll entries (requests from ::1; applies to Apache Logs)
-          --verbose                    Inform about progress (prints to STDERR)
-      -v, --version                    Prints version information
-      -h, --help                       Prints this help
-
-  This is version 1.5.2
-
-  Output formats
-  apache parsing can produce the following outputs:
-    - sqlite
-    - html
-    - txt
-  rails parsing can produce the following outputs:
-    - sqlite
-    - html
-    - txt
+        --title=TITLE                Title to use in the report
+    -f, --input-format=FORMAT        Input format (either rails or apache)
+    -i, --input-files=file,file,     Input files (can also be passed directly)
+    -t, --output-format=FORMAT       Output format: html, org, txt, sqlite.
+    -o, --output-file=OUTPUT_FILE    Output file
+    -b, --begin=DATE                 Consider entries after or on DATE
+    -e, --end=DATE                   Consider entries before or on DATE
+    -l, --limit=N                    Limit to the N most requested resources (defaults to 100)
+    -w, --width=WIDTH                Maximum width of long columns in textual reports
+    -r, --rows=ROWS                  Maximum number of rows for columns with multiple entries in textual reports
+    -p, --pattern=PATTERN            Pattern to use with ufw report to decide IP to blacklist
+    -c, --crawlers=POLICY            Decide what to do with crawlers (applies to Apache Logs)
+        --no-selfpolls               Ignore self poll entries (requests from ::1; applies to Apache Logs)
+    -n, --no-geog                    Do not geolocate entries
+        --verbose                    Inform about progress (output to STDERR)
+    -v, --version                    Prints version information
+    -h, --help                       Prints this help
+
+  This is version 1.6.0
+
+  Output formats:
+
+  - rails: txt, html, sqlite3, ufw
+  - apache: txt, html, sqlite3, ufw
   #+end_example
 
 Examples:
@@ -112,6 +141,51 @@ log_sense -f apache -i access.log -t txt > access-data.txt
 log_sense -f rails -i production.log -t html -o performance.html
 #+end_example
 
+* Code Structure
+
+The code implements a pipeline, with the following steps:
+
+  1. *Parser:* parses a log to a SQLite3 database. The database
+     contains a table with a list of events, and, in the case of Rails
+     report, a table with the errors.
+  2. *Aggregator:* takes as input a SQLite DB and aggregates data,
+      typically performing "group by", which are simpler to generate in
+      Ruby, rather than in SQL.  The module outputs a Hash, with
+      different reporting data.
+  3. *GeoLocator:* add country information to all the reporting data
+      which has an IP as one the fields.
+  4. *Shaper:* makes (geolocated) aggregated data (e.g. Hashes and
+      such), into Array of Arrays, simplifying the structure of the code
+      building the reports.
+  5. *Emitter* generates reports from shaped data using ERB.
+
+The architecture and the structure of the code is far from being nice,
+for historical reason and for a bunch of small differences existing
+between the input and the outputs to be generated.  This usually ends
+up with modifications to the code that have to be replicated in
+different parts of the code and in interferences.
+
+Among the points I would like to address:
+
+- The execution pipeline in the main script has a few exceptions to
+  manage SQLite reading/dumping and ufw report.  A linear structure
+  would be a lot nicer.
+- Two different classes are defined for steps 1, 2, and 4, to manage,
+  respectively, Apache and Rails logs.  These classes inherit from a
+  common ancestor (e.g. ApacheParser and RailsParser both inherit from
+  Parser), but there is still too little code shared.  A nicer
+  approach would be that of identifying a common DB structure and
+  unify the pipeline up to (or including) the generation of
+  reports. There are a bunch of small different things to highlight in
+  reports, which still make this difficult.  For instance, the country
+  report for Apache reports size of TX data, which is not available
+  for Rail reports.
+- Geolocation could become a lot more efficient if performed in
+  SQLite, rather than in Ruby
+- The distinction between Aggregation, Shaping, and Emission is a too
+  fine-grained and it would be nice to be able to cleanly remove one
+  of the steps.
+
 
 * Change Log
 

data/Rakefile

@@ -9,18 +9,18 @@ end
 require_relative './lib/log_sense/ip_locator.rb'
 
 desc "Convert Geolocation DB to sqlite"
-task :dbip_to_sqlite3, [:year_month] do |tasks, args|
-  filename = "./ip_locations/dbip-country-lite-#{args[:year_month]}.csv"
+task :dbip, [:filename] do |tasks, args|
+  filename = args[:filename]
 
   if !File.exist? filename
     puts "Error. Could not find: #{filename}"
     puts
     puts 'I see the following files:'
     puts Dir.glob("ip_locations/dbip-country-lite*").map { |x| "- #{x}\n" }
-    puts ''
-    puts '1. Download (if necessary) a more recent version from: https://db-ip.com/db/download/ip-to-country-lite'
-    puts '2. Save downloaded file to ip_locations/'
-    puts '3. Relaunch with YYYY-MM'
+    puts
+    puts "1. Download (if necessary) a more recent version from: https://db-ip.com/db/download/ip-to-country-lite"
+    puts "2. Save downloaded file to ip_locations/"
+    puts "3. Relaunch with YYYY-MM"
 
     exit
   else

data/exe/log_sense

@@ -1,82 +1,153 @@
 #!/usr/bin/env ruby
 
-require 'log_sense.rb'
+require "log_sense"
+require "sqlite3"
 
 #
 # Parse Command Line Arguments
 #
 
 # this better be here... OptionsParser consumes ARGV
-@command_line = ARGV.join(' ')
-@options     = LogSense::OptionsParser.parse ARGV
-@output_file = @options[:output_file]
+@command_line = ARGV.join(" ")
+@options = LogSense::OptionsParser.parse ARGV
+@input_filenames = @options[:input_filenames] + ARGV
+@output_filename = @options[:output_filename]
 
 #
-# Input files can be gotten from an option and from what remains in
-# ARGV
+# Check correctness of input data.
+#
+
+#
+# Check input files
 #
-@input_filenames = @options[:input_filenames] + ARGV
 @non_existing = @input_filenames.reject { |x| File.exist?(x) }
 
-unless @non_existing.empty?
-  $stderr.puts "Error: input file(s) '#{@non_existing.join(', ')}' do not exist"
+if @non_existing.any?
+  warn "Error: some input file(s) \"#{@non_existing.join(", ")}\" do not exist"
+  exit 1
+end
+
+#
+# Special condition: sqlite3 requires a single file as input
+#
+if @input_filenames.size > 0 &&
+   File.extname(@input_filenames.first) == "sqlite3" &&
+   @input_filenames.size > 1
+  warn "Error: you can pass only one sqlite3 file as input"
+  exit 1
+end
+
+#
+# Supported input/output chains
+#
+iformat = @options[:input_format]
+oformat = @options[:output_format]
+
+if !LogSense::OptionsChecker::compatible?(iformat, oformat)
+  warn "Error: don't know how to make #{iformat} into #{oformat}."
+  warn "Possible transformation chains:"
+  warn LogSense::OptionsChecker.chains_to_s
   exit 1
 end
-@input_files = @input_filenames.empty? ? [$stdin] : @input_filenames.map { |x| File.open(x, 'r') }
 
 #
-# Parse Log and Track Statistics
+# Do the work
 #
 
 @started_at = Time.now
 
-case @options[:input_format]
-when 'apache'
-  parser_klass = LogSense::ApacheLogParser
-  cruncher_klass = LogSense::ApacheDataCruncher
-when 'rails'
-  parser_klass = LogSense::RailsLogParser
-  cruncher_klass = LogSense::RailsDataCruncher
+if @input_filenames.size > 0 &&
+   File.extname(@input_filenames.first) == ".sqlite3"
+  warn "Reading SQLite3 DB ..." if @options[:verbose]
+  @db = SQLite3::Database.open @input_filenames.first
 else
-  $stderr.puts "Error: input format #{@options[:input_format]} not understood."
-  exit 1
+  warn "Parsing ..." if @options[:verbose]
+  @input_files = if @input_filenames.empty?
+                   [$stdin]
+                 else
+                   @input_filenames.map { |fname| File.open(fname, "r") }
+                 end
+  class_name = "LogSense::#{@options[:input_format].capitalize}LogParser"
+  parser_class = Object.const_get class_name
+  parser = parser_class.new
+  @db = parser.parse @input_files
 end
 
-$stderr.puts "Parsing input files..." if @options[:verbose]
-@db = parser_klass.parse @input_files
+if @options[:output_format] == "sqlite3"
+  warn "Saving SQLite3 DB ..." if @options[:verbose]
 
-if @options[:output_format] == 'sqlite'
-  $stderr.puts "Saving to SQLite3..." if @options[:verbose]
-  ddb = SQLite3::Database.new(@output_file || 'db.sqlite3')
-  b = SQLite3::Backup.new(ddb, 'main', @db, 'main')
+  ddb = SQLite3::Database.new(@output_filename || "db.sqlite3")
+  b = SQLite3::Backup.new(ddb, "main", @db, "main")
   b.step(-1) #=> DONE
   b.finish
+
+  exit 0
+elsif @options[:output_format] == "ufw"
+  pattern = @options[:pattern] || "php"
+
+  if @options[:input_format] == "rails"
+    query = "select distinct event.ip,event.url
+                    from error join event
+                    where event.log_id = error.log_id and
+                          event.url like '%#{pattern}%'"
+  else
+    query = "select distinct ip,path from logline
+                    where path like '%#{pattern}%'"
+  end
+
+  ips = @db.execute query
+  ips_and_urls = ips.group_by { |x| x[0] }.transform_values { |x|
+    x.map { |y| y[1..-1] }.flatten
+  }
+  ips_and_urls.each do |ip, urls|
+    puts "# #{urls[0..10].uniq.join(' ')}"
+    puts "ufw deny from #{ip}"
+    puts
+  end
+
+  exit 0
 else
-  $stderr.puts "Aggregating data..." if @options[:verbose]
-  @data = cruncher_klass.crunch @db, @options
+  warn "Aggregating data ..." if @options[:verbose]
+  class_name = "LogSense::#{@options[:input_format].capitalize}Aggregator"
+  aggr_class = Object.const_get class_name
+  aggr = aggr_class.new(@db, @options)
+  @data = aggr.aggregate
 
-  $stderr.puts "Geolocating..." if @options[:verbose]
-  @data = LogSense::IpLocator.geolocate @data
+  if @options[:geolocation]
+    warn "Geolocating ..." if @options[:verbose]
+    @data = LogSense::IpLocator.geolocate @data
 
-  $stderr.puts "Grouping by country..." if @options[:verbose]
-  country_col = @data[:ips][0].size - 1
-  @data[:countries] = @data[:ips].group_by { |x| x[country_col] }
+    warn "Grouping IPs by country ..." if @options[:verbose]
+    country_col = @data[:ips][0].size - 1
+    @data[:countries] = @data[:ips].group_by { |x| x[country_col] }
+  else
+    @data[:countries] = {}
+  end
 
   @ended_at = Time.now
   @duration = @ended_at - @started_at
 
   @data = @data.merge({
                         command: @command_line,
-                        filenames: ARGV,
+                        filenames: @input_filenames,
                         log_files: @input_files,
                         started_at: @started_at,
                         ended_at: @ended_at,
                         duration: @duration,
                         width: @options[:width]
                       })
-  #
-  # Emit Output
-  #
-  $stderr.puts "Emitting..." if @options[:verbose]
-  puts LogSense::Emitter.emit @data, @options
+
+  if @options[:verbose]
+    warn "I have the following keys in data: "
+    warn @data.keys.sort.map { |key| "#{key}: #{@data[key].class}" }.join("\n")
+  end
+
+  warn "Shaping data for output ..." if @options[:verbose]
+  class_name = "LogSense::#{@options[:input_format].capitalize}ReportShaper"
+  shaper_class = Object.const_get class_name
+  shaper = shaper_class.new
+  @reports = shaper.shape @data
+
+  warn "Emitting..." if @options[:verbose]
+  puts LogSense::Emitter.emit @reports, @data, @options
 end

data/lib/log_sense/aggregator.rb

@@ -0,0 +1,191 @@
+module LogSense
+  class Aggregator
+    def initialize
+      # not meant to be used directly
+      raise StandardError
+    end
+
+    protected
+
+    def logged_query(query)
+      puts query
+      @db.execute query
+    end
+
+    def aggregate_log_info
+      first_day_s = @db.execute "SELECT #{@date_field} from #{@table}
+                                 where #{@date_field} not NULL
+                                 order by #{@date_field}
+                                 limit 1"
+      last_day_s  = @db.execute "SELECT #{@date_field} from #{@table}
+                                 where #{@date_field} not NULL
+                                 order by #{@date_field} desc
+                                 limit 1"
+
+      # make first and last day into dates or nil
+      @first_day = first_day_s&.first&.first ? Date.parse(first_day_s[0][0]) : nil
+      @last_day =  last_day_s&.first&.first ? Date.parse(last_day_s[0][0]) : nil
+
+      @total_days = 0
+      @total_days = (@last_day - @first_day).to_i if @first_day && @last_day
+
+      evs = @db.execute "SELECT count(#{@date_field}) from #{@table}"
+      @events_in_log = @log_size = evs[0][0]
+
+      evs = @db.execute "SELECT count(#{@date_field}) from #{@table} where #{filter}"
+      @events = evs[0][0]
+
+      @source_files = @db.execute "SELECT distinct(source_file) from #{@table}"
+
+      tuv = @db.execute "SELECT count(distinct(unique_visitor)) from #{@table}
+                         where #{filter}"
+      @total_unique_visits = tuv[0][0]
+
+      @first_day_requested = @options[:from_date]
+      @last_day_requested = @options[:to_date]
+
+      @first_day_in_analysis = date_sel @first_day_requested, @first_day, :max
+      @last_day_in_analysis = date_sel @last_day_requested, @last_day, :min
+
+      @total_days_in_analysis = 0
+      if @first_day_in_analysis && @last_day_in_analysis
+        diff = (@last_day_in_analysis - @first_day_in_analysis).to_i
+        @total_days_in_analysis = diff
+      end
+    end
+
+    def aggregate_statuses
+      @statuses = @db.execute %(SELECT status, count(status) from #{@table}
+                                where #{filter}
+                                group by status
+                                order by status)
+
+      @by_day_5xx = @db.execute status_query(5)
+      @by_day_4xx = @db.execute status_query(4)
+      @by_day_3xx = @db.execute status_query(3)
+      @by_day_2xx = @db.execute status_query(2)
+
+      all_statuses = @by_day_2xx + @by_day_3xx + @by_day_4xx + @by_day_5xx
+      @statuses_by_day = all_statuses.group_by { |x| x[0] }.to_a.map { |x|
+        [x[0], x[1].map { |y| y[1] }].flatten
+      }
+    end
+
+    def aggregate_ips
+      if @table == "LogLine"
+        extra_cols = ", count(distinct(unique_visitor)), #{human_readable_size}"
+      else
+        extra_cols = ""
+      end
+
+      @ips = @db.execute %(SELECT ip, count(ip) #{extra_cols} from #{@table}
+                                  where #{filter}
+                                  group by ip                 
+                                  order by count(ip) desc     
+                                  limit #{@options[:limit]}).gsub("\n", "")
+
+      @ips_per_hour = @db.execute ip_by_time_query("hour", "%H")
+      @ips_per_day = @db.execute ip_by_time_query("day", "%Y-%m-%d")
+      @ips_per_week = @db.execute ip_by_time_query("week", "%Y-%W")
+
+      @ips_per_day_detailed = @db.execute %(
+          SELECT ip,
+                 strftime("%Y-%m-%d", #{@date_field}) as day,
+                 #{@url_field}
+                 from #{@table}
+                 where #{filter} and ip != "" and #{@url_field} != "" and
+                       #{@date_field} != ""
+                 order by ip, #{@date_field}).gsub("\n", "")
+    end
+
+    def instance_vars_to_hash
+      data = {}
+      instance_variables.each do |variable|
+        var_as_symbol = variable.to_s[1..].to_sym
+        data[var_as_symbol] = instance_variable_get(variable)
+      end
+      data
+    end
+    
+    def human_readable_size
+      mega = 1024 * 1024
+      giga = mega * 1024
+      tera = giga * 1024
+
+      %(CASE
+         WHEN sum(size) <  1024 THEN sum(size) || ' B' 
+         WHEN sum(size) >= 1024 AND sum(size) < (#{mega})
+           THEN ROUND((CAST(sum(size) AS REAL) / 1024), 2) || ' KB' 
+         WHEN sum(size) >= (#{mega}) AND sum(size) < (#{giga})
+           THEN ROUND((CAST(sum(size) AS REAL) / (#{mega})), 2) || ' MB'
+         WHEN sum(size) >= (#{giga}) AND sum(size) < (#{tera})
+           THEN ROUND((CAST(sum(size) AS REAL) / (#{giga})), 2) || ' GB'
+         WHEN sum(size) >= (#{tera})
+           THEN ROUND((CAST(sum(size) AS REAL) / (#{tera})), 2) || ' TB' 
+      END AS size).gsub("\n", "")
+    end
+
+    def human_readable_day
+      %(case cast (strftime('%w', #{@date_field}) as integer)
+          when 0 then 'Sunday'
+          when 1 then 'Monday'
+          when 2 then 'Tuesday'
+          when 3 then 'Wednesday'
+          when 4 then 'Thursday'
+          when 5 then 'Friday'
+          when 6 then 'Saturday'
+          else 'not specified'
+        end as dow).gsub("\n", "")
+    end
+
+    #
+    # generate the where clause corresponding to the command line options to filter data
+    #
+    def filter
+      from = @options[:from_date]
+      to = @options[:to_date]
+                      
+      [
+        (from ? "date(#{@date_field}) >= '#{from}'" : nil),
+        (to ? "date(#{@date_field}) <= '#{to}'" : nil),
+        (@options[:only_crawlers] ? "bot == 1" : nil),
+        (@options[:ignore_crawlers] ? "bot == 0" : nil),
+        (@options[:no_selfpolls] ? "ip != '::1'" : nil),
+        "true"
+      ].compact.join " and "
+    end
+
+    private
+
+    # given 5 builds the query to get all lines with status 5xx
+    def status_query(status)
+      %(SELECT date(#{@date_field}), count(#{@date_field}) from #{@table}
+               where substr(status, 1,1) == '#{status}' and #{filter}
+               group by date(#{@date_field})).gsub("\n", "")
+    end
+
+    # given format string, group ip by time formatted with format string
+    # (e.g. by hour if format string is "%H")
+    # name is used to give the name to the column with formatted time
+    def ip_by_time_query(name, format_string)
+      %(SELECT ip,
+               strftime("%H", #{@date_field}) as #{name},
+               count(#{@url_field}) from #{@table}
+               where #{filter} and ip != "" and
+               #{@url_field} != "" and
+               #{@date_field} != ""
+               group by ip, #{name}
+               order by ip, #{@date_field}).gsub("\n", "")
+    end
+
+    def date_sel(date1, date2, method)
+      if date1 && date2
+        [date1, date2].send(method)
+      elsif date1
+        date1
+      else
+        date2
+      end
+    end
+  end
+end