log_sense 1.5.2 → 1.6.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5f37b2247014af9bccfb8bdd205b54eb51cbef35495904444765648a96e0b9ac
-  data.tar.gz: a9cd03afb6770bf854791b8ec93e87d09532828d894b28f5dde670f1af1b1b1d
+  metadata.gz: 0a915af429fe2492f17bc767c86767e38d37a674af6206fb3b2c0a76e4cad620
+  data.tar.gz: bf4cc3a1e438b752c9c99fee5f91c85abd38de95dac947c2382e2f9825b51467
 SHA512:
-  metadata.gz: 62981216e38b92e1c3ea227726dccd592c441de32519a4df6f39bac127f55da32e82b4a1a14897b09b7032c7b3f9506a14e80e49dc43bfe8c9d86bfaa340da82
-  data.tar.gz: e5e6180008a12561d688668cffb2ef3d885d7733cf877aafaed0397b9fa0e91dd12a2998b19006dad4fe6b202bd203a00757f86d5d37efee777dbc7be43e5ab1
+  metadata.gz: 5612ef5474aa397132527588d289d9d1eba4f8954b92553e32fd5b856f2bd5441ba09d89d1bf12a0f220c602dcd2e73de2d6e360b6177b162d57adc2726442c9
+  data.tar.gz: c3b546cc177a3364b1b513f4274c1521aa1669bb17b142eb453ba73251f1e3458e7b9d1703b692ef275a474821ce7375e387155f63e3e7d5d7c9c42e1c50a150

data/CHANGELOG.org

@@ -2,6 +2,37 @@
 #+AUTHOR: Adolfo Villafiorita
 #+STARTUP: showall
 
+* 1.6.1
+
+- Country DB now stores country name.
+
+* 1.6.0
+
+- [User] New output format =ufw= generates directives to blacklist IPs
+  requesting URLs matching a pattern. For users of the Uncomplicated
+  Firewall.
+- [User] new option =--no-geo= skips geolocation, which is terribly
+  costly in the current implementation.
+- [User] Updated DB-IP country file to Dec 2022 version.
+- [User] Changed name of SQLite output format to sqlite3
+- [User] It is now possible to start analysis from a sqlite3 DB
+  generated by log_sense, breaking parsing and generation in two
+  steps.
+- [User] Check for correctness of I/O formats before launching
+  analysis
+- [User] Streak report has been renames Session.  Limited the number
+  of URLs shown in each session, to avoid buffer?/memory overflows
+  when an IP requests a massive amount of URLs.
+- [User] Added an IP-per-hour visits report.
+- [Code] A rather extensive refactoring of the source code to 
+  remove code duplications and improve code structure. 
+- [Code] Rubocop-ped various files
+- [Code] Added text renderer to DataTable, which sanitizes input and
+  further reduces risks of XSS and log poisoning attacks
+- [Code] CDN links have been ported into the Emitter module and used
+  in the Embedded Ruby Templates (erbs).  This simplifies version
+  updates of Javascript libraries used in reports.
+
 * 1.5.2
 
 - [User] Updated DB-IP country file.

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    log_sense (1.5.2)
+    log_sense (1.5.3)
       browser
       ipaddr
       iso_country_codes
@@ -16,18 +16,20 @@ GEM
       irb (>= 1.3.6)
       reline (>= 0.3.1)
     io-console (0.5.11)
-    ipaddr (1.2.4)
+    ipaddr (1.2.5)
     irb (1.4.1)
       reline (>= 0.3.0)
     iso_country_codes (0.7.8)
+    mini_portile2 (2.8.0)
     minitest (5.15.0)
     rake (12.3.3)
     reline (0.3.1)
       io-console (~> 0.5)
-    sqlite3 (1.4.4)
+    sqlite3 (1.5.4)
+      mini_portile2 (~> 2.8.0)
     terminal-table (3.0.2)
       unicode-display_width (>= 1.1.1, < 3)
-    unicode-display_width (2.2.0)
+    unicode-display_width (2.3.0)
 
 PLATFORMS
   ruby

data/README.org

@@ -9,7 +9,7 @@ Rails logs.  Written in Ruby, it runs from the command line, it is
 fast, and it can be installed on any system with a relatively recent
 version of Ruby.  We tested on Ruby 2.6.9, Ruby 3.0.x and later.
 
-LogSense reports the following data:
+When generating reports, LogSense reports the following data:
 
 - Visitors, hits, unique visitors, bandwidth used
 - Most accessed HTML pages
@@ -22,18 +22,49 @@ LogSense reports the following data:
 - IP Country location, thanks to the DP-IP lite country DB
 - Streaks: resources accessed by a given IP over time
 - Performance of Rails requests
+
+A special output format =ufw= generates rules for the [[https://launchpad.net/ufw][Uncomplicated
+Firewall]] to blacklist IPs requesting URLs matching a specific pattern.
  
 Filters from the command line allow to analyze specific periods and
 distinguish traffic generated by self polls and crawlers.
 
-LogSense generates HTML, txt, and SQLite outputs.
+LogSense generates HTML, txt, ufw, and SQLite outputs.
 
-And, of course, the compulsory screenshot:
+** Apache Report Structure
 
 #+ATTR_HTML: :width 80%
-[[file:./apache-screenshot.png]]
+[[file:./screenshots/apache-screenshot.png]]
+
+
+** Rails Report Structure
+
+#+ATTR_HTML: :width 80%
+[[file:./screenshots/rails-screenshot.png]]
+
+
+** UFW Report
 
+The output format =ufw= generates directives for Uncomplicated
+Firewall blacklisting IPs requesting URLs matching a given pattern.
 
+We use it to blacklist IPs requesting WordPress login pages on our
+websites... since we don't use WordPress for our websites.
+
+*Example*
+
+#+begin_src 
+$ log_sense -f apache -t ufw -i apache.log
+# /users/sign_in/xmlrpc.php?rsd
+ufw deny from 20.212.3.206
+
+# /wp-login.php /wordpress/wp-login.php /blog/wp-login.php /wp/wp-login.php
+ufw deny from 185.255.134.18
+
+...
+#+end_src
+
+   
 * An important word of warning
 
 [[https://owasp.org/www-community/attacks/Log_Injection][Log poisoning]] is a technique whereby attackers send requests with invalidated
@@ -48,9 +79,10 @@ opened or code executed.
 * Motivation
 
 LogSense moves along the lines of tools such as [[https://goaccess.io/][GoAccess]] (which
-strongly inspired the development of Log Sense) and [[https://umami.is/][Umami]], focusing on
-*privacy* and *data-ownership*: the data generated by LogSense is
-stored on your computer and owned by you (like it should be)[fn:1].
+strongly inspired the development of Log Sense) and [[https://umami.is/][Umami]], both
+focusing on *privacy* and *data-ownership*: the data generated by
+LogSense is stored on your computer and owned by you (like it should
+be)[fn:1].
 
 LogSense is also inspired by *static websites generators*: statistics
 are generated from the command line and accessed as static HTML files.
@@ -76,33 +108,30 @@ generated files are then made available on a private area on the web.
   #+RESULTS:
   #+begin_example
   Usage: log_sense [options] [logfile ...]
-          --title=TITLE                Title to use in the report
-      -f, --input-format=FORMAT        Input format (either rails or apache)
-      -i, --input-files=file,file,     Input files (can also be passed directly)
-      -t, --output-format=FORMAT       Output format: html, org, txt, sqlite. See below for available formats
-      -o, --output-file=OUTPUT_FILE    Output file
-      -b, --begin=DATE                 Consider entries after or on DATE
-      -e, --end=DATE                   Consider entries before or on DATE
-      -l, --limit=N                    Limit to the N most requested resources (defaults to 100)
-      -w, --width=WIDTH                Maximum width of long columns in textual reports
-      -r, --rows=ROWS                  Maximum number of rows for columns with multiple entries in textual reports
-      -c, --crawlers=POLICY            Decide what to do with crawlers (applies to Apache Logs)
-      -n, --no-selfpolls               Ignore self poll entries (requests from ::1; applies to Apache Logs)
-          --verbose                    Inform about progress (prints to STDERR)
-      -v, --version                    Prints version information
-      -h, --help                       Prints this help
-
-  This is version 1.5.2
-
-  Output formats
-  apache parsing can produce the following outputs:
-    - sqlite
-    - html
-    - txt
-  rails parsing can produce the following outputs:
-    - sqlite
-    - html
-    - txt
+        --title=TITLE                Title to use in the report
+    -f, --input-format=FORMAT        Input format (either rails or apache)
+    -i, --input-files=file,file,     Input files (can also be passed directly)
+    -t, --output-format=FORMAT       Output format: html, org, txt, sqlite.
+    -o, --output-file=OUTPUT_FILE    Output file
+    -b, --begin=DATE                 Consider entries after or on DATE
+    -e, --end=DATE                   Consider entries before or on DATE
+    -l, --limit=N                    Limit to the N most requested resources (defaults to 100)
+    -w, --width=WIDTH                Maximum width of long columns in textual reports
+    -r, --rows=ROWS                  Maximum number of rows for columns with multiple entries in textual reports
+    -p, --pattern=PATTERN            Pattern to use with ufw report to decide IP to blacklist
+    -c, --crawlers=POLICY            Decide what to do with crawlers (applies to Apache Logs)
+        --no-selfpolls               Ignore self poll entries (requests from ::1; applies to Apache Logs)
+    -n, --no-geog                    Do not geolocate entries
+        --verbose                    Inform about progress (output to STDERR)
+    -v, --version                    Prints version information
+    -h, --help                       Prints this help
+
+  This is version 1.6.0
+
+  Output formats:
+
+  - rails: txt, html, sqlite3, ufw
+  - apache: txt, html, sqlite3, ufw
   #+end_example
 
 Examples:
@@ -112,6 +141,51 @@ log_sense -f apache -i access.log -t txt > access-data.txt
 log_sense -f rails -i production.log -t html -o performance.html
 #+end_example
 
+* Code Structure
+
+The code implements a pipeline, with the following steps:
+
+  1. *Parser:* parses a log to a SQLite3 database. The database
+     contains a table with a list of events, and, in the case of Rails
+     report, a table with the errors.
+  2. *Aggregator:* takes as input a SQLite DB and aggregates data,
+      typically performing "group by", which are simpler to generate in
+      Ruby, rather than in SQL.  The module outputs a Hash, with
+      different reporting data.
+  3. *GeoLocator:* add country information to all the reporting data
+      which has an IP as one the fields.
+  4. *Shaper:* makes (geolocated) aggregated data (e.g. Hashes and
+      such), into Array of Arrays, simplifying the structure of the code
+      building the reports.
+  5. *Emitter* generates reports from shaped data using ERB.
+
+The architecture and the structure of the code is far from being nice,
+for historical reason and for a bunch of small differences existing
+between the input and the outputs to be generated.  This usually ends
+up with modifications to the code that have to be replicated in
+different parts of the code and in interferences.
+
+Among the points I would like to address:
+
+- The execution pipeline in the main script has a few exceptions to
+  manage SQLite reading/dumping and ufw report.  A linear structure
+  would be a lot nicer.
+- Two different classes are defined for steps 1, 2, and 4, to manage,
+  respectively, Apache and Rails logs.  These classes inherit from a
+  common ancestor (e.g. ApacheParser and RailsParser both inherit from
+  Parser), but there is still too little code shared.  A nicer
+  approach would be that of identifying a common DB structure and
+  unify the pipeline up to (or including) the generation of
+  reports. There are a bunch of small different things to highlight in
+  reports, which still make this difficult.  For instance, the country
+  report for Apache reports size of TX data, which is not available
+  for Rail reports.
+- Geolocation could become a lot more efficient if performed in
+  SQLite, rather than in Ruby
+- The distinction between Aggregation, Shaping, and Emission is a too
+  fine-grained and it would be nice to be able to cleanly remove one
+  of the steps.
+
 
 * Change Log
 

data/Rakefile

@@ -9,18 +9,38 @@ end
 require_relative './lib/log_sense/ip_locator.rb'
 
 desc "Convert Geolocation DB to sqlite"
-task :dbip_to_sqlite3, [:year_month] do |tasks, args|
-  filename = "./ip_locations/dbip-country-lite-#{args[:year_month]}.csv"
+task :dbip, [:filename] do |tasks, args|
+  filename_or_yyyy_mm = args[:filename]
+
+  filename = if /\d{4}-\d{2}/.match(filename_or_yyyy_mm)
+               "ip_locations/dbip-country-lite-#{filename_or_yyyy_mm}.csv"
+             else
+               filename_or_yyyy_mm
+             end
+
+  # if the filename has a .gz extension or a gzipped version of the file 
+  # exists, gunzip it
+  if File.extname(filename) == ".gz" || File.exist?("#{filename}.gz")
+    system "gunzip #{filename}.gz"
+  end
 
   if !File.exist? filename
-    puts "Error. Could not find: #{filename}"
-    puts
-    puts 'I see the following files:'
-    puts Dir.glob("ip_locations/dbip-country-lite*").map { |x| "- #{x}\n" }
-    puts ''
-    puts '1. Download (if necessary) a more recent version from: https://db-ip.com/db/download/ip-to-country-lite'
-    puts '2. Save downloaded file to ip_locations/'
-    puts '3. Relaunch with YYYY-MM'
+    puts <<-EOS
+Error. Could not find: #{filename}
+
+I see the following files:
+
+#{Dir.glob("ip_locations/dbip-country-lite*").map { |x| "- #{x}" }.join("\n")}
+
+1. Download (if necessary) a more recent version from: https://db-ip.com/db/download/ip-to-country-lite
+2. Save downloaded file to ip_locations/
+3. Relaunch with YYYY-MM (will build: dbip-country-lite-YYYY-MM.csv)
+   or with filename.
+
+Remark. If the filename has the extension .gz or if the
+filename does not exist, but a file with the same name and .gz extension
+exists, it is gunzipped first
+    EOS
 
     exit
   else

data/exe/log_sense

@@ -1,82 +1,153 @@
 #!/usr/bin/env ruby
 
-require 'log_sense.rb'
+require "log_sense"
+require "sqlite3"
 
 #
 # Parse Command Line Arguments
 #
 
 # this better be here... OptionsParser consumes ARGV
-@command_line = ARGV.join(' ')
-@options     = LogSense::OptionsParser.parse ARGV
-@output_file = @options[:output_file]
+@command_line = ARGV.join(" ")
+@options = LogSense::OptionsParser.parse ARGV
+@input_filenames = @options[:input_filenames] + ARGV
+@output_filename = @options[:output_filename]
 
 #
-# Input files can be gotten from an option and from what remains in
-# ARGV
+# Check correctness of input data.
+#
+
+#
+# Check input files
 #
-@input_filenames = @options[:input_filenames] + ARGV
 @non_existing = @input_filenames.reject { |x| File.exist?(x) }
 
-unless @non_existing.empty?
-  $stderr.puts "Error: input file(s) '#{@non_existing.join(', ')}' do not exist"
+if @non_existing.any?
+  warn "Error: some input file(s) \"#{@non_existing.join(", ")}\" do not exist"
+  exit 1
+end
+
+#
+# Special condition: sqlite3 requires a single file as input
+#
+if @input_filenames.size > 0 &&
+   File.extname(@input_filenames.first) == "sqlite3" &&
+   @input_filenames.size > 1
+  warn "Error: you can pass only one sqlite3 file as input"
+  exit 1
+end
+
+#
+# Supported input/output chains
+#
+iformat = @options[:input_format]
+oformat = @options[:output_format]
+
+if !LogSense::OptionsChecker::compatible?(iformat, oformat)
+  warn "Error: don't know how to make #{iformat} into #{oformat}."
+  warn "Possible transformation chains:"
+  warn LogSense::OptionsChecker.chains_to_s
   exit 1
 end
-@input_files = @input_filenames.empty? ? [$stdin] : @input_filenames.map { |x| File.open(x, 'r') }
 
 #
-# Parse Log and Track Statistics
+# Do the work
 #
 
 @started_at = Time.now
 
-case @options[:input_format]
-when 'apache'
-  parser_klass = LogSense::ApacheLogParser
-  cruncher_klass = LogSense::ApacheDataCruncher
-when 'rails'
-  parser_klass = LogSense::RailsLogParser
-  cruncher_klass = LogSense::RailsDataCruncher
+if @input_filenames.size > 0 &&
+   File.extname(@input_filenames.first) == ".sqlite3"
+  warn "Reading SQLite3 DB ..." if @options[:verbose]
+  @db = SQLite3::Database.open @input_filenames.first
 else
-  $stderr.puts "Error: input format #{@options[:input_format]} not understood."
-  exit 1
+  warn "Parsing ..." if @options[:verbose]
+  @input_files = if @input_filenames.empty?
+                   [$stdin]
+                 else
+                   @input_filenames.map { |fname| File.open(fname, "r") }
+                 end
+  class_name = "LogSense::#{@options[:input_format].capitalize}LogParser"
+  parser_class = Object.const_get class_name
+  parser = parser_class.new
+  @db = parser.parse @input_files
 end
 
-$stderr.puts "Parsing input files..." if @options[:verbose]
-@db = parser_klass.parse @input_files
+if @options[:output_format] == "sqlite3"
+  warn "Saving SQLite3 DB ..." if @options[:verbose]
 
-if @options[:output_format] == 'sqlite'
-  $stderr.puts "Saving to SQLite3..." if @options[:verbose]
-  ddb = SQLite3::Database.new(@output_file || 'db.sqlite3')
-  b = SQLite3::Backup.new(ddb, 'main', @db, 'main')
+  ddb = SQLite3::Database.new(@output_filename || "db.sqlite3")
+  b = SQLite3::Backup.new(ddb, "main", @db, "main")
   b.step(-1) #=> DONE
   b.finish
+
+  exit 0
+elsif @options[:output_format] == "ufw"
+  pattern = @options[:pattern] || "php"
+
+  if @options[:input_format] == "rails"
+    query = "select distinct event.ip,event.url
+                    from error join event
+                    where event.log_id = error.log_id and
+                          event.url like '%#{pattern}%'"
+  else
+    query = "select distinct ip,path from logline
+                    where path like '%#{pattern}%'"
+  end
+
+  ips = @db.execute query
+  ips_and_urls = ips.group_by { |x| x[0] }.transform_values { |x|
+    x.map { |y| y[1..-1] }.flatten
+  }
+  ips_and_urls.each do |ip, urls|
+    puts "# #{urls[0..10].uniq.join(' ')}"
+    puts "ufw deny from #{ip}"
+    puts
+  end
+
+  exit 0
 else
-  $stderr.puts "Aggregating data..." if @options[:verbose]
-  @data = cruncher_klass.crunch @db, @options
+  warn "Aggregating data ..." if @options[:verbose]
+  class_name = "LogSense::#{@options[:input_format].capitalize}Aggregator"
+  aggr_class = Object.const_get class_name
+  aggr = aggr_class.new(@db, @options)
+  @data = aggr.aggregate
 
-  $stderr.puts "Geolocating..." if @options[:verbose]
-  @data = LogSense::IpLocator.geolocate @data
+  if @options[:geolocation]
+    warn "Geolocating ..." if @options[:verbose]
+    @data = LogSense::IpLocator.geolocate @data
 
-  $stderr.puts "Grouping by country..." if @options[:verbose]
-  country_col = @data[:ips][0].size - 1
-  @data[:countries] = @data[:ips].group_by { |x| x[country_col] }
+    warn "Grouping IPs by country ..." if @options[:verbose]
+    country_col = @data[:ips][0].size - 1
+    @data[:countries] = @data[:ips].group_by { |x| x[country_col] }
+  else
+    @data[:countries] = {}
+  end
 
   @ended_at = Time.now
   @duration = @ended_at - @started_at
 
   @data = @data.merge({
                         command: @command_line,
-                        filenames: ARGV,
+                        filenames: @input_filenames,
                         log_files: @input_files,
                         started_at: @started_at,
                         ended_at: @ended_at,
                         duration: @duration,
                         width: @options[:width]
                       })
-  #
-  # Emit Output
-  #
-  $stderr.puts "Emitting..." if @options[:verbose]
-  puts LogSense::Emitter.emit @data, @options
+
+  if @options[:verbose]
+    warn "I have the following keys in data: "
+    warn @data.keys.sort.map { |key| "#{key}: #{@data[key].class}" }.join("\n")
+  end
+
+  warn "Shaping data for output ..." if @options[:verbose]
+  class_name = "LogSense::#{@options[:input_format].capitalize}ReportShaper"
+  shaper_class = Object.const_get class_name
+  shaper = shaper_class.new
+  @reports = shaper.shape @data
+
+  warn "Emitting..." if @options[:verbose]
+  puts LogSense::Emitter.emit @reports, @data, @options
 end