footfall 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: cd91000a047937d1598600d2b06cf063db586d468841895158055b2d7a520f7f
+  data.tar.gz: 6e5615676b235a42b9ca63bc9bb3d801e710c4e821c8dfe31424c5f0c199928a
+SHA512:
+  metadata.gz: 577732a8ccaad0e9d2a942a3e6f2cf7866399e2381fc3ce48d256d9eb3be5a24a3bcf2b1bb68d52979137f9294693006fe8fd0e2c3fdb1d10419590329503958
+  data.tar.gz: cebb8fd907bb3f3556dfcebc0000781ccb2f3a226e9c5def5722e007f0ea451feda02700e8c2f31587dd37129baa0f43bd4e3bbfb969d8ee2e8410767af44e61

data/CHANGELOG.md

@@ -0,0 +1,9 @@
+# Changelog
+
+## 0.1.0 — 2026-10-09
+
+- First version, extracted from mini-audition.com's own counter: the middleware, the
+  SQLite schema and the terminal report. Options for where downloads live, the site's
+  host, a proxy's client address header and the bot pattern; the report's download columns
+  and feed pattern are configurable. It never changes a table it didn't make: give it a
+  file of its own.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Oleh Khomei
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,102 @@
+# footfall
+
+Cookieless visitor, page view and download counts for a Rack app, in a SQLite file. Footfall
+is what a shop counts at its door: people coming in, not who they are.
+
+- **Pages, visitors, referrers and downloads.** A page view is a GET of HTML (or an Atom or
+  RSS feed) answered with 200; a download, a GET under `/download/` that sends the whole
+  file, counted once even when it's resumed in pieces.
+- **Nobody can be picked out.** A visitor is a hash of their address and browser, salted
+  anew every day; the salt is forgotten the next day, so a day's visitors can be counted but
+  not followed from day to day, or traced back. No cookies, no IP addresses, and only the
+  site a visitor came from, not the full referrer.
+- **Bots left out:** crawlers, link previews, monitors and command-line tools.
+- **One file, no service.** SQLite in WAL mode, next to your app; a report for the terminal.
+- **Never breaks a page.** A counting error is logged and the response goes out as it was.
+
+## Install
+
+```ruby
+# Gemfile
+gem "footfall"
+```
+
+## Count
+
+```ruby
+# config.ru
+require "footfall"
+
+# Outermost, so it sees every response, downloads too. Without a path it counts nothing,
+# which suits development and tests.
+use Footfall::Middleware, path: ENV["FOOTFALL_PATH"]
+```
+
+Options:
+
+| Option | Default | |
+|---|---|---|
+| `path:` | (required) | The SQLite file, made with its folder; `nil` turns counting off. |
+| `downloads:` | `"/download/"` | The path prefix of files counted as downloads; `nil` for none. |
+| `host:` | the request's host | This site, whose own links aren't referrers (`www.` is ignored). |
+| `client_ip_header:` | none | The header a proxy puts the visitor's address in, such as Cloudflare's `"CF-Connecting-IP"`. Only behind such a proxy: anyone can send it. |
+| `bots:` | `Footfall::Middleware::BOTS` | A regexp of user agents left out. |
+
+In Rails, `config.middleware.insert_before 0, Footfall::Middleware, path: …`.
+
+## Report
+
+```ruby
+require "footfall"
+
+Footfall::Report.new("storage/footfall.sqlite3",
+  title: "example.com",
+  downloads: { "App downloads" => "%.dmg", "Plugin installs" => "Plugin-%" }
+).print(days: 30)
+```
+
+```
+example.com, the last 30 days (since 2026-09-10); bots left out
+
+Totals
+  Visitors (a day each)  Page views  App downloads  Plugin installs
+  ---------------------  ----------  -------------  ---------------
+                    412         903             57               12
+
+By day
+  Day         Visitors  Views  App downloads
+  …
+```
+
+Then the busiest pages, the sites visitors came from, every download, and the feed's
+readers on the latest day. `downloads:` names a column in the totals per kind of file, as
+SQL `LIKE` patterns of the file names; the first is also the day by day column. Feeds
+(`feeds:`, `"%/feed.xml"` by default) are counted apart from pages, since readers poll them.
+
+A small script makes it a command, for example for `kamal app exec`:
+
+```ruby
+#!/usr/bin/env ruby
+require "bundler/setup"
+require "footfall"
+
+Footfall::Report.new(ENV.fetch("FOOTFALL_PATH"), title: "example.com")
+  .print(days: Integer(ARGV.first || 30))
+```
+
+## The data
+
+Three tables, for queries of your own:
+
+- `views`: `day`, `path`, `referrer` (a host, or `""`), `visitor`, `count`, one row per day,
+  page, referrer and visitor;
+- `downloads`: `day`, `file`, `visitor`, `count`;
+- `salts`: today's salt only.
+
+Footfall only creates tables that aren't there and never alters existing ones; give it a
+file of its own. Days are in the server's time zone. Visitors are only comparable within a day: count
+`DISTINCT day || visitor` for "visitors, a day each".
+
+## License
+
+MIT

data/lib/footfall/database.rb

@@ -0,0 +1,32 @@
+require "fileutils"
+require "sqlite3"
+
+module Footfall
+  # The SQLite file: a row per day, page, referring site and visitor for page views, and per
+  # day, file and visitor for downloads, each with how many times; and the day's salt.
+  module Database
+    SCHEMA = <<~SQL
+      CREATE TABLE IF NOT EXISTS salts (day TEXT PRIMARY KEY, salt TEXT NOT NULL);
+      CREATE TABLE IF NOT EXISTS views (
+        day TEXT NOT NULL, path TEXT NOT NULL, referrer TEXT NOT NULL,
+        visitor TEXT NOT NULL, count INTEGER NOT NULL DEFAULT 1,
+        PRIMARY KEY (day, path, referrer, visitor));
+      CREATE TABLE IF NOT EXISTS downloads (
+        day TEXT NOT NULL, file TEXT NOT NULL, visitor TEXT NOT NULL,
+        count INTEGER NOT NULL DEFAULT 1,
+        PRIMARY KEY (day, file, visitor));
+    SQL
+
+    # For counting: made (with its folder) if it isn't there yet. Readonly for reports.
+    def self.open(path, readonly: false)
+      return SQLite3::Database.new(path, readonly: true) if readonly
+
+      FileUtils.mkdir_p(File.dirname(path))
+      db = SQLite3::Database.new(path)
+      db.busy_timeout = 2_000
+      db.execute("PRAGMA journal_mode = WAL")
+      db.execute_batch(SCHEMA)
+      db
+    end
+  end
+end

data/lib/footfall/middleware.rb

@@ -0,0 +1,118 @@
+require "date"
+require "digest"
+require "rack"
+require "securerandom"
+require "uri"
+
+module Footfall
+  # Counts page views and downloads, after the app has answered. A page view is a GET of
+  # a page (HTML, or an Atom or RSS feed) answered with 200; a download, a GET under
+  # `downloads` that sends the whole file (200, or 206 from its first byte), so redirects
+  # and the later pieces of a resumed download aren't counted again.
+  #
+  # Nothing that identifies anyone is kept: a visitor is a hash of their IP address and
+  # browser with a salt that's new every day and forgotten the next. No cookies, no
+  # addresses, no full referrers (only the site a visitor came from). Bots are left out.
+  class Middleware
+    # Crawlers, link previews, monitors and command-line tools.
+    BOTS = /bot|crawl|spider|slurp|preview|monitor|curl|wget|python|ruby|go-http|java|
+      httpclient|headless|lighthouse|facebookexternalhit|embedly|whatsapp|telegram/xi
+
+    PAGES = %r{\A(text/html|application/(atom|rss)\+xml)\b}
+
+    # `path`: the SQLite file; nil turns counting off.
+    # `downloads`: the path prefix of files to count as downloads; nil for none.
+    # `host`: this site, whose own links aren't counted as referrers (the request's host
+    #   unless given).
+    # `client_ip_header`: the header a proxy in front puts the visitor's address in, such
+    #   as Cloudflare's "CF-Connecting-IP"; without one, the request's own address. Only
+    #   set it behind such a proxy: anyone can send the header (it only skews the counts).
+    def initialize(app, path:, downloads: "/download/", host: nil, client_ip_header: nil,
+      bots: BOTS, today: -> { Date.today })
+      @app = app
+      @downloads = downloads
+      @host = host && site(host)
+      @client_ip = client_ip_header && "HTTP_#{client_ip_header.upcase.tr("-", "_")}"
+      @bots = bots
+      @today = today
+      @lock = Mutex.new
+      @db = path && Database.open(path)
+    end
+
+    def call(env)
+      status, headers, body = @app.call(env)
+      record(Rack::Request.new(env), status, headers) if @db
+      [ status, headers, body ]
+    end
+
+    private
+
+    # Never lets counting break a page.
+    def record(request, status, headers)
+      return unless request.get?
+      agent = request.user_agent.to_s
+      return if agent.empty? || @bots.match?(agent)
+
+      path = request.path_info
+      if @downloads && path.start_with?(@downloads)
+        download(request, path, status)
+      elsif status == 200 && PAGES.match?(headers["content-type"].to_s)
+        upsert("views", day: today, path: path, referrer: referrer(request),
+          visitor: visitor(request))
+      end
+    rescue StandardError => error
+      warn "footfall: #{error.class}: #{error.message}"
+    end
+
+    def download(request, path, status)
+      range = request.get_header("HTTP_RANGE").to_s
+      return unless status == 200 || (status == 206 && range.match?(/\Abytes=0-/))
+
+      upsert("downloads", day: today, file: File.basename(path), visitor: visitor(request))
+    end
+
+    # The site a visitor came from ("news.ycombinator.com"), or "" for none or this one.
+    def referrer(request)
+      from = site(URI.parse(request.referer.to_s).host.to_s)
+      from == (@host || site(request.host.to_s)) ? "" : from
+    rescue URI::InvalidURIError
+      ""
+    end
+
+    def site(host) = host.downcase.delete_prefix("www.")
+
+    def visitor(request)
+      address = (@client_ip && request.get_header(@client_ip)) || request.ip
+      Digest::SHA256.hexdigest("#{salt}|#{address}|#{request.user_agent}")[0, 16]
+    end
+
+    def today = @today.call.iso8601
+
+    # Today's salt, made the first time it's needed; the days before are forgotten then.
+    def salt
+      day = today
+      return @salt if @salt_day == day
+
+      @lock.synchronize do
+        @db.execute("DELETE FROM salts WHERE day < ?", [ day ])
+        @db.execute("INSERT OR IGNORE INTO salts (day, salt) VALUES (?, ?)",
+          [ day, SecureRandom.hex(16) ])
+        @salt = @db.get_first_value("SELECT salt FROM salts WHERE day = ?", [ day ])
+        @salt_day = day
+      end
+      @salt
+    end
+
+    def upsert(table, values)
+      # As text: Rack's paths and headers can be binary strings, which SQLite would keep
+      # as blobs, never equal to text in a query.
+      values = values.transform_values { |value| value.to_s.dup.force_encoding(Encoding::UTF_8).scrub }
+      marks = ([ "?" ] * values.size).join(", ")
+      @lock.synchronize do
+        @db.execute(
+          "INSERT INTO #{table} (#{values.keys.join(", ")}) VALUES (#{marks}) " \
+          "ON CONFLICT DO UPDATE SET count = count + 1", values.values)
+      end
+    end
+  end
+end

data/lib/footfall/report.rb

@@ -0,0 +1,104 @@
+require "date"
+
+module Footfall
+  # The counts in the terminal: totals, the last two weeks day by day, the busiest pages,
+  # the sites visitors came from, every download, and feed readers. Feeds are counted
+  # apart from pages, since readers poll them.
+  #
+  #   Footfall::Report.new("stats.sqlite3", title: "example.com",
+  #     downloads: { "App" => "%.dmg", "Plugins" => "%.zip" }).print(days: 30)
+  class Report
+    # `downloads`: a column in the totals per kind of file, as SQL LIKE patterns of their
+    # names; the first is also the day by day column.
+    # `feeds`: a LIKE pattern of the feeds' paths.
+    def initialize(path, title: nil, downloads: { "Downloads" => "%" }, feeds: "%/feed.xml",
+      io: $stdout)
+      @db = Database.open(path, readonly: true)
+      @title = title
+      @downloads = downloads
+      @feeds = feeds
+      @io = io
+    end
+
+    def print(days: 30, today: Date.today)
+      since = (today - days + 1).iso8601
+      @io.puts [ @title, "the last #{days} days (since #{since}); bots left out" ].compact.join(", ")
+      totals(since)
+      by_day(since)
+      pages(since)
+      referrers(since)
+      downloads
+      @io.puts "\nFeed readers (the latest day): #{feed_readers || 0}"
+    end
+
+    private
+
+    def totals(since)
+      visitors, views = @db.get_first_row(<<~SQL, [ since, @feeds ])
+        SELECT COUNT(DISTINCT day || visitor), COALESCE(SUM(count), 0)
+        FROM views WHERE day >= ? AND path NOT LIKE ?
+      SQL
+      files = @downloads.values.map do |pattern|
+        @db.get_first_value(
+          "SELECT COUNT(*) FROM downloads WHERE day >= ? AND file LIKE ?", [ since, pattern ])
+      end
+      table("Totals", [ [ visitors, views, *files ] ],
+        [ "Visitors (a day each)", "Page views", *@downloads.keys ])
+    end
+
+    def by_day(since)
+      rows = @db.execute(<<~SQL, [ @downloads.values.first, since, @feeds ])
+        SELECT day, COUNT(DISTINCT visitor), SUM(count),
+          (SELECT COUNT(*) FROM downloads d WHERE d.day = views.day AND d.file LIKE ?)
+        FROM views WHERE day >= ? AND path NOT LIKE ?
+        GROUP BY day ORDER BY day DESC LIMIT 14
+      SQL
+      table("By day", rows, [ "Day", "Visitors", "Views", @downloads.keys.first ])
+    end
+
+    def pages(since)
+      rows = @db.execute(<<~SQL, [ since, @feeds ])
+        SELECT path, COUNT(DISTINCT day || visitor), SUM(count)
+        FROM views WHERE day >= ? AND path NOT LIKE ?
+        GROUP BY path ORDER BY 2 DESC LIMIT 15
+      SQL
+      table("Pages", rows, [ "Page", "Visitors", "Views" ])
+    end
+
+    def referrers(since)
+      rows = @db.execute(<<~SQL, [ since, @feeds ])
+        SELECT referrer, COUNT(DISTINCT day || visitor)
+        FROM views WHERE day >= ? AND referrer != '' AND path NOT LIKE ?
+        GROUP BY referrer ORDER BY 2 DESC LIMIT 15
+      SQL
+      table("Where they came from", rows, [ "Site", "Visitors" ])
+    end
+
+    def downloads
+      rows = @db.execute(<<~SQL)
+        SELECT file, COUNT(*), MAX(day) FROM downloads GROUP BY file ORDER BY MAX(day) DESC, file
+      SQL
+      table("Downloads (all time)", rows, [ "File", "Downloads", "Last" ])
+    end
+
+    def feed_readers
+      @db.get_first_value(<<~SQL, [ @feeds, @feeds ])
+        SELECT COUNT(DISTINCT visitor) FROM views
+        WHERE path LIKE ? AND day = (SELECT MAX(day) FROM views WHERE path LIKE ?)
+      SQL
+    end
+
+    def table(title, rows, headers)
+      @io.puts "\n#{title}"
+      return @io.puts("  (none)") if rows.empty?
+
+      all = [ headers ] + rows.map { |row| row.map(&:to_s) }
+      widths = headers.each_index.map { |i| all.map { |row| row[i].length }.max }
+      all.each_with_index do |row, index|
+        cells = row.each_with_index.map { |cell, i| i.zero? ? cell.ljust(widths[i]) : cell.rjust(widths[i]) }
+        @io.puts "  #{cells.join("  ")}"
+        @io.puts "  #{widths.map { |width| "-" * width }.join("  ")}" if index.zero?
+      end
+    end
+  end
+end

data/lib/footfall/version.rb

@@ -0,0 +1,3 @@
+module Footfall
+  VERSION = "0.1.0"
+end

data/lib/footfall.rb

@@ -0,0 +1,9 @@
+require "footfall/version"
+require "footfall/database"
+require "footfall/middleware"
+require "footfall/report"
+
+# Cookieless visitor, page view and download counts for a Rack app, in a SQLite file:
+# Footfall::Middleware counts them, Footfall::Report prints them.
+module Footfall
+end

metadata

@@ -0,0 +1,126 @@
+--- !ruby/object:Gem::Specification
+name: footfall
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Oleh Khomei
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rack
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '2.2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '2.2'
+- !ruby/object:Gem::Dependency
+  name: sqlite3
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rack-test
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: |
+  Footfall is what a shop counts at its door: people coming in, not who they
+  are. Rack middleware that counts page views, visitors, the sites they came
+  from and file downloads into a SQLite file, and a report for the terminal.
+  No cookies, no IP addresses, no full referrers: a visitor is a hash salted
+  anew every day and the salt forgotten the next, so a day's visitors can be
+  counted but not followed from day to day or traced back. Bots are left out.
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE
+- README.md
+- lib/footfall.rb
+- lib/footfall/database.rb
+- lib/footfall/middleware.rb
+- lib/footfall/report.rb
+- lib/footfall/version.rb
+homepage: https://github.com/varyform/footfall
+licenses:
+- MIT
+metadata:
+  source_code_uri: https://github.com/varyform/footfall
+  changelog_uri: https://github.com/varyform/footfall/blob/main/CHANGELOG.md
+  rubygems_mfa_required: 'true'
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.2'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.20
+specification_version: 4
+summary: Cookieless visitor, page view and download counts for a Rack app, in a SQLite
+  file.
+test_files: []