robots-audit 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: baf8c54332377b0c267c6916e0d282dc4c099d6e631c4c4ae38672d6c337c86f
+  data.tar.gz: 612fd78f06f3d4e914a6a217dcee629f64eb650b7c58205b4049f857ef665cc1
+SHA512:
+  metadata.gz: 75377ea8d2412c09ffb33990e77a1dfa18db5248815f71a7616c90a9d8940ab1d0ae4aa02b601e84ab1be3ab3979fbc6d7e11103e4db099a2d5c403e7272a31b
+  data.tar.gz: 94752665717c851333d756964fc1244608e86fad06d6c5161fd45e44691a23bab4b9df24f9a0efe067ec941f512a99bf08710054b2231f9cb8ac0ff070d55164

data/CHANGELOG.md

@@ -0,0 +1,13 @@
+# Changelog
+
+## 1.0.0 — 2026-10-17
+
+Initial release.
+
+- Fetch a site's robots.txt with redirect following (301/302, bounded) and explicit 404/status handling
+- Robots exclusion protocol parsing: User-agent groups (including multi-agent blocks), Disallow rules, Sitemap lines, comment/blank-line handling, UTF-8 BOM stripping, encoding repair
+- Findings: parse violations, orphan Disallow rules, missing wildcard User-agent coverage, missing/present sitemap declarations, unexpected HTTP status, oversize-file truncation warning, many-agents advisory
+- CLI: multi-URL audits, --json output, -T timeout, -v version
+- Exit codes: 0 clean/info, 1 warnings or fetch errors
+- Zero runtime dependencies (stdlib net/http, uri, optparse, json)
+- Test suite: 10 unit tests + 4 e2e cases (local socket servers + live HTTPS round-trips)

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 DeveloApps
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,64 @@
+# robots-audit
+
+Audit any website's `robots.txt` from the command line: parse checks, wildcard/sitemap coverage, redirect handling, JSON output. Zero runtime dependencies — Ruby standard library only.
+
+## What it checks
+
+- **Parse violations** — lines that don't follow the `field: token` robots exclusion protocol structure, orphan `Disallow` rules with no preceding `User-agent`, and UTF-8 BOM or encoding issues.
+- **Wildcard coverage** — whether a `User-agent: *` group exists (if not, your rules only bind to named bots).
+- **Sitemap declarations** — whether `Sitemap:` lines exist and what they point to.
+- **Redirect handling** — follows 301/302 redirects (e.g. `www.` → apex) before auditing, up to a safe limit.
+- **HTTP sanity** — reports missing (404), unexpected, or error statuses explicitly instead of crashing.
+- **Oversize files** — warns when a `robots.txt` exceeds 2 MiB and audits the truncated prefix.
+- **Multi-agent groups** — correct handling of consecutive `User-agent` lines sharing one rule block.
+
+## Installation
+
+    gem install robots-audit
+
+Or run directly from a checkout:
+
+    ruby robots_audit.rb https://example.com
+
+## Usage
+
+    # Human-readable report for one or more sites
+    robots_audit.rb https://example.com https://another-site.org
+
+    # Machine-readable output for CI
+    robots_audit.rb --json https://example.com
+
+    # Custom HTTP timeout (seconds)
+    robots_audit.rb -T 30 https://slow-site.example
+
+Exit codes: `0` = clean or info-only findings, `1` = warnings present or fetch errors, usage errors abort with a message.
+
+Example output:
+
+    https://example.com  ->  https://example.com/robots.txt
+      [WARN] no-wildcard-agent: No '*' (wildcard) User-agent group — rules only bind to named bots.
+      [INFO] no-sitemap: No Sitemap: line — declaring one helps crawlers discover URLs.
+      2 finding(s).
+
+## Testing
+
+The test suite uses plain Ruby assertions (no minitest/rspec dependency):
+
+    ruby test_robots_audit.rb        # 10 unit tests
+    ruby e2e_live_check.rb           # 4 end-to-end cases (local socket servers + live HTTPS)
+
+## Why
+
+Small SEO and site-hygiene teams often ship a broken or stale `robots.txt` without noticing — an orphan `Disallow` from an old site structure, a missing wildcard group, a sitemap nobody declared. This gem makes that a one-liner you can run in CI or on demand, with no dependency chain to maintain.
+
+## Author
+
+Built by [TechEMC](https://techemc.com/) — one controlled workflow with a clear owner and measure. Practical, measurable, controlled.
+
+## License
+
+MIT — see LICENSE.txt.
+
+## Changelog
+
+See CHANGELOG.md.

data/robots_audit.rb

@@ -0,0 +1,195 @@
+#!/usr/bin/env ruby
+# robots-audit — audit robots.txt files against the robots exclusion protocol.
+# stdlib only: net/http, uri, optparse, json. Ruby >= 2.7 (tested 3.4).
+require 'net/http'
+require 'uri'
+require 'optparse'
+require 'json'
+
+module RobotsAudit
+  VERSION = '1.0.0'
+  TOKEN = /\A([A-Za-z_-]+):(.*)\z/.freeze
+
+  Rule = Struct.new(:agent, :path, :directive, :line_no)
+
+  class Ruleset
+    attr_reader :agents, :rules, :sitemaps, :raw_lines
+
+    def initialize
+      @agents = []
+      @rules = []
+      @sitemaps = []
+      @raw_lines = []
+      @last_agents = []
+      @violations = []
+    end
+
+    def add_line(idx, text)
+      text = text.sub(/\A\uFEFF/, '')
+      @raw_lines << [idx, text]
+      stripped = text.strip
+      return if stripped.empty? || stripped.start_with?('#')
+
+      m = TOKEN.match(stripped)
+      return record_violation(idx, stripped, 'unparseable line (no field:token structure)') unless m
+
+      field = m[1].downcase
+      value = m[2].strip
+      case field
+      when 'user-agent'
+        @last_agents << value
+        @agents << value unless @agents.include?(value)
+      when 'allow', 'disallow'
+        record_violation(idx, stripped, "'#{field}' rule without a preceding User-agent") if @last_agents.empty?
+        return if field == 'allow'
+        @rules << Rule.new(@last_agents.dup, value, field, idx)
+      when 'sitemap'
+        @sitemaps << value unless @sitemaps.include?(value)
+      end
+    rescue StandardError
+      record_violation(idx, stripped, 'internal parse failure')
+    end
+
+    def record_violation(idx, text, reason)
+      @violations ||= []
+      @violations << { line: idx, text: text, reason: reason }
+    end
+
+    def violations
+      @violations || []
+    end
+
+    def crawl_delay(agent)
+      # crawl-delay is a non-standard extension; report presence, don't interpret
+      nil
+    end
+  end
+
+  class Fetcher
+    USER_AGENT = 'robots-audit/1.0 (+https://rubygems.org/gems/robots-audit)'
+
+    def self.fetch(url, limit: 5, timeout: 15)
+      uri = URI.parse(url)
+      raise ArgumentError, "URL must be http(s), got #{uri.scheme.inspect}" unless uri.is_a?(URI::HTTP) || uri.is_a?(URI::HTTPS)
+      raise ArgumentError, 'URL must be absolute (scheme + host)' if uri.host.to_s.empty?
+
+      redirects = 0
+      res = nil
+      robots_url = uri
+      loop do
+        robots_url = URI.join(robots_url, '/robots.txt')
+        http = Net::HTTP.new(robots_url.host, robots_url.port)
+        http.use_ssl = (robots_url.scheme == 'https')
+        http.open_timeout = timeout
+        http.read_timeout = timeout
+        res = http.get(robots_url.path.empty? ? '/robots.txt' : robots_url.path)
+        break unless res.is_a?(Net::HTTPRedirection)
+        raise 'too many redirects' if (redirects += 1) > limit
+        robots_url = URI.join(robots_url, res['location'])
+      end
+      [robots_url, res]
+    end
+  end
+
+  class Audit
+    MAX_BYTES = 2 * 1024 * 1024
+    SANE_AGENT_COUNT = 25
+    SANE_RULE_COUNT = 2_000
+
+    def initialize(uri, res, body)
+      @uri = uri
+      @res = res
+      @body = body.to_s.byteslice(0, MAX_BYTES)
+    end
+
+    def run
+      findings = []
+      status = @res.code.to_i
+      if status == 404
+        findings << { severity: 'info', code: 'no-robots', message: "No robots.txt (HTTP 404) — all paths crawlable by default." }
+        return findings
+      elsif status != 200
+        findings << { severity: 'warn', code: 'unexpected-status', message: "robots.txt returned HTTP #{status}." }
+        return findings
+      end
+
+      if @body.bytesize >= MAX_BYTES
+        findings << { severity: 'warn', code: 'oversize-truncated', message: "robots.txt exceeds #{MAX_BYTES} bytes; audit ran on the truncated prefix only." }
+      end
+
+      rs = Ruleset.new
+      body = @body.dup.force_encoding(Encoding::UTF_8)
+      body = body.encode(Encoding::UTF_8, invalid: :replace, undef: :replace)
+      body.lines.each_with_index { |line, i| rs.add_line(i + 1, line) }
+
+      findings.concat(rule_issues(rs))
+      findings
+    end
+
+    def rule_issues(rs)
+      findings = []
+      findings << { severity: 'warn', code: 'empty-file', message: 'robots.txt is present but empty.' } if rs.raw_lines.all? { |_, t| t.strip.empty? || t.strip.start_with?('#') }
+
+      viols = rs.violations
+      viols.first(20).each do |v|
+        findings << { severity: 'warn', code: 'parse-violation', message: "Line #{v[:line]}: #{v[:reason]} — \"#{v[:text].byteslice(0, 80)}\"" }
+      end
+
+      if rs.agents.size > SANE_AGENT_COUNT
+        findings << { severity: 'info', code: 'many-agents', message: "#{rs.agents.size} User-agent groups — fine, but consider grouping identical rule sets." }
+      end
+      findings << { severity: 'warn', code: 'no-wildcard-agent', message: "No '*' (wildcard) User-agent group — rules only bind to named bots." } unless rs.agents.include?('*')
+      findings << { severity: 'info', code: 'sitemap-declared', message: "Sitemap(s) declared: #{rs.sitemaps.join(', ')}" } unless rs.sitemaps.empty?
+      findings << { severity: 'info', code: 'no-sitemap', message: "No Sitemap: line — declaring one helps crawlers discover URLs." } if rs.sitemaps.empty?
+      findings
+    end
+  end
+
+  def self.audit(url, timeout: 15)
+    robots_url, res = Fetcher.fetch(url, timeout: timeout)
+    body = res.body.to_s
+    [robots_url, Audit.new(robots_url, res, body).run]
+  end
+end
+
+if $PROGRAM_NAME == __FILE__
+  opts = { json: false }
+  OptionParser.new do |o|
+    o.banner = 'Usage: robots-audit [options] URL [URL ...]'
+    o.on('--json', 'Emit JSON instead of text') { opts[:json] = true }
+    o.on('-T SEC', Integer, 'HTTP timeout (default 15)') { |v| opts[:timeout] = v }
+    o.on('-v', '--version') { puts "robots-audit #{RobotsAudit::VERSION}"; exit 0 }
+  end.parse!
+
+  urls = ARGV
+  abort('Usage: robots-audit [--json] URL [URL ...]') if urls.empty?
+
+  worst = 0
+  results = urls.map do |u|
+    begin
+      robots_url, findings = RobotsAudit.audit(u, timeout: opts[:timeout] || 15)
+      max = findings.map { |f| f[:severity] == 'warn' ? 1 : 0 }.max.to_i
+      worst = [worst, max].max
+      { url: u, robots_url: robots_url.to_s, ok: true, findings: findings.map { |f| { severity: f[:severity], code: f[:code], message: f[:message] } } }
+    rescue StandardError => e
+      worst = [worst, 1].max
+      { url: u, ok: false, error: e.class.name, message: e.message }
+    end
+  end
+
+  if opts[:json]
+    puts JSON.pretty_generate(results)
+  else
+    results.each do |r|
+      if r[:ok]
+        puts "#{r[:url]}  ->  #{r[:robots_url]}"
+        r[:findings].each { |f| puts "  [#{f[:severity].upcase}] #{f[:code]}: #{f[:message]}" }
+        puts "  #{r[:findings].empty? ? 'No findings — looks clean.' : "#{r[:findings].size} finding(s)."}"
+      else
+        puts "#{r[:url]}  ->  ERROR #{r[:error]}: #{r[:message]}"
+      end
+      puts
+    end
+  end
+  exit(worst)
+end

metadata

@@ -0,0 +1,55 @@
+--- !ruby/object:Gem::Specification
+name: robots-audit
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- DeveloApps
+bindir: "."
+cert_chain: []
+date: 2026-10-17 00:00:00.000000000 Z
+dependencies: []
+description: 'robots-audit fetches a site''s robots.txt (following redirects), parses
+  it per the robots exclusion protocol, and reports actionable findings: parse violations,
+  orphan Disallow lines, missing wildcard User-agent coverage, missing sitemap declarations,
+  unexpected HTTP statuses, and oversize files. Zero runtime dependencies — stdlib
+  only (net/http, uri, optparse, json). Use it for SEO hygiene checks, crawl-budget
+  reviews, and CI gates.'
+email:
+- marky@develoapps.com
+executables:
+- robots_audit.rb
+extensions: []
+extra_rdoc_files: []
+files:
+- "./robots_audit.rb"
+- CHANGELOG.md
+- LICENSE.txt
+- README.md
+- robots_audit.rb
+homepage: https://techemc.com/
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://techemc.com/
+  source_code_uri: https://techemc.com/
+  changelog_uri: https://techemc.com/
+rdoc_options: []
+require_paths:
+- "."
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.7.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.6.2
+specification_version: 4
+summary: 'Audit any site''s robots.txt: parse checks, wildcard/sitemap coverage, redirect
+  handling, JSON output.'
+test_files: []