log_sense 1.3.1 → 1.3.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7e05abbc9fc93aa9ccd057f3a01f6c93d966e964695d486e3425b3b4bc073c52
-  data.tar.gz: 7514a501cf9ac3766b457bca1d31a545a1331e35d5e8591aa81965994943580d
+  metadata.gz: 6498507bb6ae7e6ba5505609421154358790f095bd9cfe5ab912920b2409f7d1
+  data.tar.gz: 0ccb5fa005b8f1e47545bf98ce05273f07bd452d3be54c5bd1a7f124c96558d9
 SHA512:
-  metadata.gz: 91cbf90786d0c6479b246b9df92f67fcae73491d3d622eb99c364a8d279aa1ca8aac009ba765c1b75734ca9d664d334bd4aed970c02b6e8fda8a3a15f8534c52
-  data.tar.gz: 9b41aa16dd595fe664f6a1ba90bbb196dcbc945c760f6ea6ddcb30569ff0767f70f06487790c3eda0f0d8cd35c30cba4bab2982a3825067025e30a5a5dcb56f3
+  metadata.gz: e12760ccdfc518c7221afa337f4ac1c8a77219cd3ed65510486e3ce945e6129519aa3fe93b05845a5430b9bc52359d70591e0fcbd346e971527af1760c304e5c
+  data.tar.gz: 8d38d8ed84287722b77f51599598b84a2257f9f86d2dd3815faf25dfb7db4ab16e68dbf737c7d005519ac5f9fe1d4b41cd896dfaf862eabb6a7dc4f033d2bb13

data/CHANGELOG.org

@@ -2,13 +2,20 @@
 #+AUTHOR: Adolfo Villafiorita
 #+STARTUP: showall
 
-* Changes in log_sense 1.1.2
-  <2021-12-17 Fri>
+* 1.3.2
 
-- Added Rails Log HTML output
+- [Code] HTML reports now generate JSON data which is shared between
+  DataTable and Vega Light: this should reduce page size and loading
+  time of HTML reports
+- [Doc] Added screenshot and fixed some text
+- [Doc] Fixes requirements on Ruby version
 
-* Changes in log_sense 1.1.1 and earlier
-  <2021-12-17 Fri>
+* 1.3.1
 
-- In the Git commit messages (not very informative, I am afraid).
+- [Code] Strengthened parsing of Apache Logs (added WebDav and other methods)
 
+* 1.3.0
+
+- [Code] Removed dependency from =apache_log-parser= and implemented our own
+  parser for the combined format.
+  

data/Gemfile.lock

@@ -1,8 +1,7 @@
 PATH
   remote: .
   specs:
-    log_sense (1.0.7)
-      apache_log-parser
+    log_sense (1.3.1)
       browser
       ipaddr
       iso_country_codes
@@ -12,7 +11,6 @@ PATH
 GEM
   remote: https://rubygems.org/
   specs:
-    apache_log-parser (3.1.2)
     browser (5.3.1)
     byebug (11.1.3)
     ipaddr (1.2.3)

data/README.org

@@ -4,24 +4,10 @@
 
 * Introduction
 
-LogSense generates reports and statistics from Apache web logs in the
-=combined= format and from Rails logs.  Written in Ruby, it runs from
-the command line, it is fast, and it can be installed on any system
-which supports Ruby.
-
-LogSense moves along the lines of tools such as [[https://goaccess.io/][GoAccess]]
-and [[https://umami.is/][Umami]], focusing on privacy and data-ownership: the data
-generated by LogSense is stored on your computer and owned by
-you (like it should be).
-
-LogSense is also inspired by *static websites generators*: statistics
-are generated from the command line and accessed as static HTML files.
-By generating static resources, LogSense significantly reduces the
-attack surface of your webserver and installation headaches.
-
-We have, for instance, a cron job running on our servers, generating
-statistics at night.  The generated files are then made available on a
-private area on the web.
+LogSense generates reports and statistics from Apache and Ruby on
+Rails logs.  Written in Ruby, it runs from the command line, it is
+fast, and it can be installed on any system with a relatively recent
+version of Ruby.  We tested on Ruby 2.6.9, Ruby 3.0.x and later.
 
 LogSense reports the following data:
 
@@ -40,7 +26,27 @@ LogSense reports the following data:
 Filters from the command line allow to analyze specific periods and
 distinguish traffic generated by self polls and crawlers.
 
-LogSense generates HTML, txt (Org Mode), and SQLite outputs.
+LogSense generates HTML, txt, and SQLite outputs.
+
+And, of course, the compulsory screenshot:
+
+#+ATTR_HTML: :width 80%
+[[file:./apache-screenshot.png]]
+
+* Motivation
+
+LogSense moves along the lines of tools such as [[https://goaccess.io/][GoAccess]] (which
+strongly inspired the development of Log Sense) and [[https://umami.is/][Umami]], focusing on
+*privacy* and *data-ownership*: the data generated by LogSense is
+stored on your computer and owned by you (like it should be)[fn:1].
+
+LogSense is also inspired by *static websites generators*: statistics
+are generated from the command line and accessed as static HTML files.
+LogSense thus significantly reduces the attack surface of your
+webserver and installation headaches.  We have, for instance, a Cron
+job running on our servers, generating statistics at night.  The
+generated files are then made available on a private area on the web.
+
 
 * Installation
 
@@ -81,37 +87,48 @@ LogSense generates HTML, txt (Org Mode), and SQLite outputs.
     - txt
   #+end_example
 
+Examples:
+
+#+begin_example sh
+log_sense -f apache -i access.log -t txt > access-data.txt
+log_sense -f rails -i production.log -t html > performance.txt
+#+end_example
+
 * Change Log
 
 See the [[file:CHANGELOG.org][CHANGELOG]] file.
 
 * Compatibility
 
-LogSense should run on any system on which Ruby runs.
+LogSense should run on any system on which a recent version of Ruby
+runs.  We tested it with Ruby 2.6.9 and Ruby 3.x.x.
 
 Concerning the outputs:
 
-- The HTML report uses [[https://picturepan2.github.io/spectre/][Spectre.css]] and (will use) [[https://vega.github.io/vega-lite/][Vega Light]], which
-  are downloaded from a CDN
-- The textual format is compatible with Org Mode and can be further
-  processed to any format Org Mode can be exported to (including HTML
-  and PDF),
+- HTML reports use [[https://get.foundation/][Zurb Foundation]], [[https://www.datatables.net/][Data Tables]], and [[https://vega.github.io/vega-lite/][Vega Light]], which
+  are all downloaded from a CDN
+- The textual format is compatible with [[https://orgmode.org/][Org Mode]] and can be further
+  processed to any format [[https://orgmode.org/][Org Mode]] can be exported to (including HTML
+  and PDF)
 
 * Author and Contributors
 
-[[http://ict4g.net/adolfo][Adolfo Villafiorita]].
+[[https://shair.tech][Shair.Tech]]
 
 * Known Bugs
 
-Some known bugs and an unknown number of unknown bugs.
-
+No known bugs; an unknown number of unknown bugs.
 (See the open issues for the known bugs.)
 
 * License
 
 Distributed under the terms of the [[http://opensource.org/licenses/MIT][MIT License]].
 
-Geolocation is made possible by the DB-IP.com IP to City database, released under
-a CC license.
-
+Geolocation is made possible by the DB-IP.com IP to City database,
+released under a CC license.
 
+[fn:1] There is a small catch: CSS and JavaScript for layout and plots
+are downloaded from a CDN.  Technically, thus, if you generate HTML
+reports and open them, a request is performed and the CDN might keep a
+track (see [[https://en.wikipedia.org/wiki/Content_delivery_network#Security_and_privacy][CDN Security and Privacy on Wikipedia]] for more details).
+Textual reports don't have this issue.

data/lib/log_sense/templates/_output_table.html.erb

@@ -4,22 +4,26 @@ def slugify string
 end
 %>
 
-<table id="<%= slugify(title || "") %>-table" class="table unstriped data-table <%= slugify(title || "") %>">
+<table id="table-<%= index %>" class="table unstriped">
   <thead>
     <tr>
-      <% header.each do |heading| %>
-        <th class="<%= slugify(heading) %>"><%= heading %></th>
+      <% report[:header].each do |header| %>
+        <th><%= header %></th>
       <% end %>
     </tr>
   </thead>
   <tbody>
-    <% rows.each do |row| %>
-      <tr>
-        <% row.each_with_index do |cell, i| %>
-          <td class="<%= slugify (header[i] || "") %>"><%= cell %></td>
-        <% end %>
-      </tr>
-    <% end %>
   </tbody>
 </table>
-
+<script>
+ $(document).ready(function(){
+   $('#table-<%= index %>').dataTable({
+     data: data_<%= index %>,
+     columns: [
+       <% report[:header].each do |header| %>
+       { data: '<%= header %>', className: '<%= slugify(header) %>' },
+       <% end %>
+     ]
+   });
+ });
+</script>

data/lib/log_sense/templates/_report_data.html.erb

@@ -0,0 +1,12 @@
+<script>
+ /* this is used both by Vega and DataTable */
+ data_<%= index %> = [
+   <% report[:rows].each do |row| %>
+   {
+     <% report[:header].each_with_index do |h, i| %>
+     "<%= h %>": <%= (row[i].class == Integer or row[i].class == Float) ? row[i] : "\"#{row[i]}\"" %>,
+     <% end %>
+   },
+   <% end %>
+ ]
+</script>

data/lib/log_sense/templates/apache.html.erb

@@ -17,6 +17,8 @@
     <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/foundation-sites@6.7.4/dist/css/foundation.min.css">
     <link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/v/zf/dt-1.11.3/datatables.min.css"/>
 
+    <script type="text/javascript" src="https://code.jquery.com/jquery-3.6.0.min.js"></script>
+    <script type="text/javascript" src="https://cdn.datatables.net/v/zf/dt-1.11.3/datatables.min.js"></script>
 
     <script src="https://cdn.jsdelivr.net/npm/vega@5.21.0"></script>
     <script src="https://cdn.jsdelivr.net/npm/vega-lite@5.2.0"></script>
@@ -426,7 +428,7 @@
                     <%= report[:title] %>
                   </h2>
                 </div>
-
+                <%= render "report_data.html.erb", report: report, index: index %>
                 <% if report[:vega_spec] %>
                   <div id="<%= "plot-#{index}" %>"></div>
                   <script>
@@ -436,22 +438,15 @@
                        width: "container",
                        description: "<%= report[:title] %>",
                        data: {
-                         values: [
-                           <% report[:rows].each do |row| %>
-                           {
-                             <% report[:header].each_with_index do |h, i| %>
-                             "<%= h %>": <%= (row[i].class == Integer or row[i].class == Float) ? row[i] : "\"#{row[i]}\"" %>,
-                             <% end %>
-                           },
-                           <% end %>
-                         ]
+                         values: data_<%= index %>
                        },
                    });
                    vegaEmbed('#<%= "plot-#{index}"%>', plot_spec_<%= index %>);
                   </script>
                 <% end %>
-                  <div class="card-section">
-                  <%= render "output_table.html.erb", report %>
+
+                <div class="card-section">
+                  <%= render "output_table.html.erb", report: report, index: index %>
                 </div>
               </article>
             <% end %>
@@ -569,9 +564,7 @@
         </section>
       </div>
 
-      <script type="text/javascript" src="https://code.jquery.com/jquery-3.6.0.min.js"></script>
       <script type="text/javascript" src="js/vendor/what-input.js"></script>
-      <script type="text/javascript" src="https://cdn.datatables.net/v/zf/dt-1.11.3/datatables.min.js"></script>
       <script src="https://cdn.jsdelivr.net/npm/vega@5"></script>    
       <script type="text/javascript" src="https://cdn.jsdelivr.net/npm/foundation-sites@6.7.4/dist/js/foundation.min.js" crossorigin="anonymous"></script>
       <script>

data/lib/log_sense/templates/rails.html.erb

@@ -17,6 +17,8 @@
     <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/foundation-sites@6.7.4/dist/css/foundation.min.css">
     <link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/v/zf/dt-1.11.3/datatables.min.css"/>
 
+    <script type="text/javascript" src="https://code.jquery.com/jquery-3.6.0.min.js"></script>
+    <script type="text/javascript" src="https://cdn.datatables.net/v/zf/dt-1.11.3/datatables.min.js"></script>
 
     <script src="https://cdn.jsdelivr.net/npm/vega@5.21.0"></script>
     <script src="https://cdn.jsdelivr.net/npm/vega-lite@5.2.0"></script>
@@ -341,7 +343,7 @@
                     <%= report[:title] %>
                   </h2>
                 </div>
-
+                <%= render "report_data.html.erb", report: report, index: index %>
                 <% if report[:vega_spec] %>
                   <div id="<%= "plot-#{index}" %>"></div>
                   <script>
@@ -351,22 +353,14 @@
                        width: "container",
                        description: "<%= report[:title] %>",
                        data: {
-                         values: [
-                           <% report[:rows].each do |row| %>
-                           {
-                             <% report[:header].each_with_index do |h, i| %>
-                             "<%= h %>": <%= (row[i].class == Integer or row[i].class == Float) ? row[i] : "\"#{row[i]}\"" %>,
-                             <% end %>
-                           },
-                           <% end %>
-                         ]
+                         values: data_<%= index %>
                        },
                    });
                    vegaEmbed('#<%= "plot-#{index}"%>', plot_spec_<%= index %>);
                   </script>
                 <% end %>
                 <div class="card-section">
-                  <%= render "output_table.html.erb", report %>
+                  <%= render "output_table.html.erb", report: report, index: index %>
                 </div>
               </article>
             <% end %>
@@ -392,9 +386,7 @@
         </section>
       </div>
 
-      <script type="text/javascript" src="https://code.jquery.com/jquery-3.6.0.min.js"></script>
       <script type="text/javascript" src="js/vendor/what-input.js"></script>
-      <script type="text/javascript" src="https://cdn.datatables.net/v/zf/dt-1.11.3/datatables.min.js"></script>
       <script src="https://cdn.jsdelivr.net/npm/vega@5"></script>    
       <script type="text/javascript" src="https://cdn.jsdelivr.net/npm/foundation-sites@6.7.4/dist/js/foundation.min.js" crossorigin="anonymous"></script>
       <script>

data/lib/log_sense/version.rb

@@ -1,3 +1,3 @@
 module LogSense
-  VERSION = "1.3.1"
+  VERSION = "1.3.2"
 end

data/log_sense.gemspec

@@ -10,7 +10,7 @@ Gem::Specification.new do |spec|
   spec.description   = %q{Generate analystics in HTML, txt, and SQLite format from an Apache and Rails log files.}
   spec.homepage      = "https://www.ict4g.net/gitea/adolfo/log_sense"
   spec.license       = "MIT"
-  spec.required_ruby_version = Gem::Requirement.new(">= 2.3.0")
+  spec.required_ruby_version = Gem::Requirement.new(">= 2.6.9")
 
   spec.metadata["allowed_push_host"] = "https://rubygems.org/"
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: log_sense
 version: !ruby/object:Gem::Version
-  version: 1.3.1
+  version: 1.3.2
 platform: ruby
 authors:
 - Adolfo Villafiorita
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2021-12-24 00:00:00.000000000 Z
+date: 2022-01-12 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: browser
@@ -124,6 +124,7 @@ files:
 - LICENSE.txt
 - README.org
 - Rakefile
+- apache-screenshot.png
 - bin/console
 - bin/setup
 - exe/log_sense
@@ -143,6 +144,7 @@ files:
 - lib/log_sense/templates/_output_table.html.erb
 - lib/log_sense/templates/_performance.html.erb
 - lib/log_sense/templates/_performance.txt.erb
+- lib/log_sense/templates/_report_data.html.erb
 - lib/log_sense/templates/_summary.html.erb
 - lib/log_sense/templates/_summary.txt.erb
 - lib/log_sense/templates/apache.html.erb
@@ -169,14 +171,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: 2.3.0
+      version: 2.6.9
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.2.32
+rubygems_version: 3.0.3.1
 signing_key:
 specification_version: 4
 summary: Generate analytics from an Apache and Rails log file.