crawlscope 0.7.2 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 18717d5f025a5e0dc10b3f5a231b52d788a7ad09fe017b803815dc1cf22b7f11
-  data.tar.gz: 25f80cac563922d3c34d069c34b6c5521064fb4a74dd98cc54f6bb04e6de283a
+  metadata.gz: 4b830667c3f7647baba6ca372e78daa5f4116e2178fc585aff4f387671e71b81
+  data.tar.gz: 357a5ceab12f86addda14927ba03be7357f20d301ddeb58b72a87527133653ce
 SHA512:
-  metadata.gz: 322dacf831bfa19dd16a6ebf3917e8d8bcaab462b440468945e4e9e07768a42955afe7f418b217594f9654c22490d32124f6fdb95004a20f23508f76bca6d7db
-  data.tar.gz: b301ad0f2c7c92dd3dc436f7019c62d7958bdd402eb8a887626d938a9bfd085d8d21bdab76a5243f3e2d9684d94ffce554e5108db26995d67b63140fcadbb9e2
+  metadata.gz: 954a7f1044ae369e01fc571331acbf0b9ebd47f56d8047d028c6967a85cf5caefa673a3bfa05b782a814d3233f883ea13ca98a3ab961d2e8de47d788804f6158
+  data.tar.gz: 8a59e970c7191847a25b855e7342d0ebaab7b881540d41ac78b9b558460ffd9a679fbfff3c54ef78e22f2bd287d8009f9d58961128b9c4b74ac3773ce8dc82dd

data/CHANGELOG.md

@@ -5,6 +5,24 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.8.0] - 2026-07-24
+
+
+### Added
+
+- report Server-Timing metrics
+
+- authenticate timing crawl requests
+
+
+
+
+### Fixed
+
+- use portable crawl configuration defaults
+
+
+
 ## [0.7.2] - 2026-07-23
 
 

data/README.md

@@ -93,6 +93,11 @@ crawlscope validate --url https://example.com --sitemap https://example.com/site
 
 Child sitemap indexes are supported automatically.
 
+Set `CRAWLSCOPE_PROFILE_TOKEN` through the process environment or your secret
+manager when the site protects diagnostic timing with `X-Profile-Token`.
+Crawlscope intentionally has no `--profile-token` flag because command arguments
+can be exposed through shell history and process listings.
+
 Validation output is grouped for terminal scanning:
 
 ```text
@@ -141,10 +146,69 @@ puts result.issues.to_a.map(&:message)
 - `urls`: sitemap URLs selected for validation
 - `pages`: fetched page snapshots
 - `issues`: structured issues with `code`, `severity`, `category`, `url`, and `message`
+- `server_timing_summary`: aggregate response timing data when pages publish it
 
 `result.ok?` returns `false` when an error is present. Warnings and notices
 remain available through `result.issues` without making the result fail.
 
+## Server Timing
+
+Crawlscope parses the
+[`Server-Timing`](https://www.w3.org/TR/server-timing/) response header for both
+HTTP and browser-rendered crawls. Each page exposes its parsed header through
+`page.server_timing`:
+
+```ruby
+page.server_timing.each do |metric|
+  puts [metric.name, metric.duration, metric.description].compact.join(": ")
+end
+```
+
+The validation report adds a `Server Timing` section only when at least one page
+publishes the header. It includes:
+
+- header coverage and the number of pages publishing durations
+- sample and page counts with average, p50, p95, and maximum per metric
+- non-duration signals, including cache status and routing descriptions
+- the ten pages with the largest individual metric
+- the number of malformed entries ignored during parsing
+
+`dur` values are reported as milliseconds, as recommended by the specification.
+Crawlscope does not add durations together because metrics such as `total`,
+`app`, and `db` may overlap. A page's worst-offender entry is its largest
+individual metric instead.
+
+The current HTTP and browser transports expose response headers, not response
+trailers. Metrics published only in a `Server-Timing` trailer are therefore not
+available to Crawlscope.
+
+Rails 8 applications can enable `ActionDispatch::ServerTiming` with:
+
+```ruby
+config.server_timing = true
+```
+
+Applications that expose timing only to authenticated diagnostics can set
+`config.profile_token`. When this value is present, Crawlscope sends it as
+`X-Profile-Token` on same-origin sitemap, HTTP, and browser document requests.
+The header survives same-origin redirects but is removed before a cross-origin
+redirect, and browser subresources never receive it. Keep the token in the host
+application's secret store rather than a URL, command argument, report, or
+checked-in configuration.
+
+`CRAWLSCOPE_PROFILE_TOKEN` is the portable default for Rails, the standalone
+CLI, Rake tasks, and plain Ruby callers. Rails applications may instead assign
+`config.profile_token` from encrypted credentials when that is their established
+secret-management boundary; explicit configuration takes precedence over the
+environment.
+
+New Rails applications enable it in development by default; production remains
+opt-in. Rails publishes the Active Support notification names observed during
+each request and sums repeated events with the same name. The exact metrics
+therefore depend on the application and request, but commonly include controller,
+view, database, and cache instrumentation. Crawlscope accepts Rails' dotted
+metric names and reports each metric independently.
+
 ## Rails Usage
 
 Run the install generator after adding the gem:
@@ -162,10 +226,11 @@ Customize the `Crawlscope.configure` block inside the generated initializer:
 
 ```ruby
 Crawlscope.configure do |config|
-  config.base_url = -> { "https://example.com" }
-  config.sitemap_path = -> { Rails.public_path.join("sitemap.xml").to_s }
-  config.site_name = "Example"
-  config.schema_registry = -> { Crawlscope::SchemaRegistry.default }
+  config.base_url = -> { ENV.fetch("CRAWLSCOPE_BASE_URL", "http://localhost:3000") }
+  config.sitemap_path = lambda {
+    ENV.fetch("SITEMAP", "#{config.base_url.to_s.chomp("/")}/sitemap.xml")
+  }
+  config.site_name = ENV.fetch("CRAWLSCOPE_SITE_NAME", "Application")
 end
 ```
 
@@ -193,6 +258,7 @@ Available environment overrides:
 
 - `URL`
 - `SITEMAP`
+- `CRAWLSCOPE_PROFILE_TOKEN`
 - `RULES=metadata,links`
 - `JS=1` or `RENDERER=browser`
 - `TIMEOUT=30`
@@ -226,10 +292,10 @@ bundle exec rake 'crawlscope:validate:ldjson[https://example.com/article]'
 
 `crawlscope:validate` runs all default sitemap rules: indexability, metadata,
 structured data, uniqueness, content quality, and links. `URL` is the site
-base. Without `SITEMAP`, Crawlscope uses the configured sitemap path, then
-falls back to `/sitemap.xml`. With `SITEMAP`, Crawlscope uses `URL` as the site
-base and validates URLs from that sitemap. `SITEMAP` may be a full URL or a
-local file path.
+base. Without `SITEMAP`, Crawlscope uses the configured sitemap URL, then fetches
+`/sitemap.xml` from `URL` over HTTP. With `SITEMAP`, Crawlscope uses `URL` as
+the site base and validates URLs from that sitemap. `SITEMAP` may be a full URL
+or an explicitly selected local file path.
 
 Plain `rake` does not pass `--url` style flags to tasks. Use `URL=...` or the
 task-argument form above instead.

data/lib/crawlscope/browser.rb

@@ -4,13 +4,15 @@ require "nokogiri"
 
 module Crawlscope
   class Browser
-    def initialize(base_url:, timeout_seconds:, network_idle_timeout_seconds:, scroll_page:)
+    def initialize(base_url:, timeout_seconds:, network_idle_timeout_seconds:, scroll_page:, profile_token: nil)
       @base_url = base_url
       @timeout_seconds = timeout_seconds
       @network_idle_timeout_seconds = network_idle_timeout_seconds
+      @profile_token = profile_token
       @scroll_page = scroll_page
       @browser = build_browser
       @page = @browser.create_page
+      configure_profile_requests
     end
 
     def close
@@ -72,6 +74,21 @@ module Crawlscope
       )
     end
 
+    def configure_profile_requests
+      return if @profile_token.to_s.empty?
+
+      @page.network.intercept(resource_type: :document)
+      @page.on(:request) do |request|
+        headers = RequestHeaders.add_profile_token(
+          request.headers.dup,
+          url: request.url,
+          base_url: @base_url,
+          profile_token: @profile_token
+        )
+        request.continue(headers: headers.map { |name, value| {name: name.to_s, value: value.to_s} })
+      end
+    end
+
     def scroll_for_render
       @page.evaluate("(function() { if (document.body) { window.scrollTo(0, document.body.scrollHeight); } })()")
       wait_for_network_idle

data/lib/crawlscope/configuration.rb

@@ -11,7 +11,7 @@ module Crawlscope
     RENDERERS = %i[http browser].freeze
     DEFAULT_TIMEOUT_SECONDS = 20
 
-    attr_writer :allowed_statuses, :base_url, :browser_factory, :concurrency, :fetch_executor, :network_idle_timeout_seconds, :output, :renderer, :rule_registry, :schema_registry, :scroll_page, :site_name, :sitemap_path, :timeout_seconds
+    attr_writer :allowed_statuses, :base_url, :browser_factory, :concurrency, :fetch_executor, :network_idle_timeout_seconds, :output, :profile_token, :renderer, :rule_registry, :schema_registry, :scroll_page, :site_name, :sitemap_path, :timeout_seconds
 
     def allowed_statuses
       value = resolve(@allowed_statuses)
@@ -59,6 +59,10 @@ module Crawlscope
       value.nil? ? $stdout : value
     end
 
+    def profile_token
+      resolve(@profile_token) || ENV["CRAWLSCOPE_PROFILE_TOKEN"]
+    end
+
     def renderer
       value = resolve(@renderer)
       normalized_value = value.to_s.strip
@@ -93,6 +97,7 @@ module Crawlscope
         concurrency: concurrency,
         fetch_executor: fetch_executor,
         network_idle_timeout_seconds: network_idle_timeout_seconds,
+        profile_token: profile_token,
         renderer: renderer,
         timeout_seconds: timeout_seconds,
         allowed_statuses: allowed_statuses,

data/lib/crawlscope/crawl.rb

@@ -2,7 +2,7 @@
 
 module Crawlscope
   class Crawl
-    def initialize(base_url:, sitemap_path:, rules:, schema_registry:, browser_factory: nil, concurrency: Configuration::DEFAULT_CONCURRENCY, fetch_executor: nil, network_idle_timeout_seconds: Configuration::DEFAULT_BROWSER_NETWORK_IDLE_TIMEOUT_SECONDS, renderer: :http, scroll_page: Configuration::DEFAULT_BROWSER_SCROLL_PAGE, timeout_seconds: Configuration::DEFAULT_TIMEOUT_SECONDS, allowed_statuses: Configuration::DEFAULT_ALLOWED_STATUSES)
+    def initialize(base_url:, sitemap_path:, rules:, schema_registry:, browser_factory: nil, concurrency: Configuration::DEFAULT_CONCURRENCY, fetch_executor: nil, network_idle_timeout_seconds: Configuration::DEFAULT_BROWSER_NETWORK_IDLE_TIMEOUT_SECONDS, profile_token: nil, renderer: :http, scroll_page: Configuration::DEFAULT_BROWSER_SCROLL_PAGE, timeout_seconds: Configuration::DEFAULT_TIMEOUT_SECONDS, allowed_statuses: Configuration::DEFAULT_ALLOWED_STATUSES)
       @base_url = base_url
       @sitemap_path = sitemap_path
       @rules = Array(rules)
@@ -10,6 +10,7 @@ module Crawlscope
       @browser_factory = browser_factory
       @concurrency = concurrency
       @network_idle_timeout_seconds = network_idle_timeout_seconds
+      @profile_token = profile_token
       @renderer = renderer.to_sym
       @fetch_executor = fetch_executor || default_fetch_executor
       @scroll_page = scroll_page
@@ -49,6 +50,7 @@ module Crawlscope
         adapter: http_adapter,
         concurrency: @concurrency,
         fetch_executor: @fetch_executor,
+        profile_token: @profile_token,
         timeout_seconds: @timeout_seconds
       ).urls(base_url: @base_url)
       raise ValidationError, "No URLs found in sitemap: #{@sitemap_path}" if urls.empty?
@@ -61,6 +63,7 @@ module Crawlscope
         base_url: @base_url,
         timeout_seconds: @timeout_seconds,
         network_idle_timeout_seconds: @network_idle_timeout_seconds,
+        profile_token: @profile_token,
         scroll_page: @scroll_page
       )
     rescue LoadError => error
@@ -71,7 +74,7 @@ module Crawlscope
       if @renderer == :browser
         (@browser_factory || method(:browser)).call
       else
-        Http.new(base_url: @base_url, timeout_seconds: @timeout_seconds, adapter: http_adapter)
+        Http.new(base_url: @base_url, timeout_seconds: @timeout_seconds, adapter: http_adapter, profile_token: @profile_token)
       end
     end
 

data/lib/crawlscope/http.rb

@@ -10,10 +10,11 @@ module Crawlscope
     MAX_REDIRECTS = 5
     USER_AGENT = "Mozilla/5.0 (compatible; Crawlscope/1.0)"
 
-    def initialize(base_url:, timeout_seconds:, adapter: nil)
+    def initialize(base_url:, timeout_seconds:, adapter: nil, profile_token: nil)
       @base_url = base_url
       @timeout_seconds = timeout_seconds
       @adapter = adapter
+      @profile_token = profile_token
       @connections_by_thread = Concurrent::Map.new
     end
 
@@ -28,6 +29,12 @@ module Crawlscope
     def fetch(url)
       response = connection.get(url) do |request|
         request.headers["User-Agent"] = USER_AGENT
+        RequestHeaders.add_profile_token(
+          request.headers,
+          url: url,
+          base_url: @base_url,
+          profile_token: @profile_token
+        )
       end
 
       final_url = response.env.url.to_s
@@ -67,7 +74,11 @@ module Crawlscope
     def connection
       @connections_by_thread.compute_if_absent(Thread.current.object_id) do
         Faraday.new do |faraday|
-          faraday.response :follow_redirects, limit: MAX_REDIRECTS
+          faraday.response(
+            :follow_redirects,
+            limit: MAX_REDIRECTS,
+            callback: RequestHeaders.method(:strip_profile_token_on_cross_origin_redirect)
+          )
           faraday.options.timeout = @timeout_seconds
           faraday.options.open_timeout = @timeout_seconds
           faraday.adapter @adapter if @adapter

data/lib/crawlscope/page.rb

@@ -19,5 +19,13 @@ module Crawlscope
     def html?
       !doc.nil?
     end
+
+    def header(name)
+      headers.find { |key, _value| key.to_s.casecmp?(name.to_s) }&.last
+    end
+
+    def server_timing
+      @server_timing ||= ServerTiming.new(header("server-timing"))
+    end
   end
 end

data/lib/crawlscope/reporter.rb

@@ -19,13 +19,18 @@ module Crawlscope
 
       if result.issues.size.zero?
         @io.puts("Status: OK")
-        return
+      else
+        @io.puts("Status: #{status_for(result.issues)}")
+        @io.puts("Issues: #{result.issues.size} total (#{severity_summary(result.issues)})")
       end
 
-      @io.puts("Status: #{status_for(result.issues)}")
-      @io.puts("Issues: #{result.issues.size} total (#{severity_summary(result.issues)})")
-      @io.puts("")
+      ServerTiming::Reporter.new(io: @io).report(
+        result.server_timing_summary,
+        base_url: result.base_url
+      )
+      return if result.issues.size.zero?
 
+      @io.puts("")
       report_summary(result.issues)
       @io.puts("")
       report_issue_groups(result.issues, base_url: result.base_url)

data/lib/crawlscope/request_headers.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+require "uri"
+
+module Crawlscope
+  module RequestHeaders
+    PROFILE_TOKEN = "X-Profile-Token"
+
+    module_function
+
+    def add_profile_token(headers, url:, base_url:, profile_token:)
+      return headers if profile_token.to_s.empty?
+      return headers unless same_origin?(url, base_url: base_url)
+
+      headers[PROFILE_TOKEN] = profile_token
+      headers
+    end
+
+    def strip_profile_token_on_cross_origin_redirect(response_env, request_env)
+      return if same_origin?(request_env[:url], base_url: response_env[:url])
+
+      request_env[:request_headers].delete(PROFILE_TOKEN)
+    end
+
+    def same_origin?(url, base_url:)
+      url_uri = URI.join(base_url.to_s, url.to_s)
+      base_uri = URI.parse(base_url.to_s)
+
+      [url_uri.scheme, url_uri.host, url_uri.port] == [base_uri.scheme, base_uri.host, base_uri.port]
+    rescue URI::Error
+      false
+    end
+  end
+end

data/lib/crawlscope/result.rb

@@ -5,5 +5,9 @@ module Crawlscope
     def ok?
       issues.none?(&:error?)
     end
+
+    def server_timing_summary
+      ServerTiming::Summary.new(pages)
+    end
   end
 end

data/lib/crawlscope/run.rb

@@ -44,17 +44,7 @@ module Crawlscope
       value = @configuration.sitemap_path
       return value unless value.to_s.strip.empty?
 
-      local_path = File.expand_path("public/sitemap.xml", Dir.pwd)
-      return local_path if local_path_default?(base_url: base_url) && File.exist?(local_path)
-
       "#{base_url.to_s.chomp("/")}/sitemap.xml"
     end
-
-    def local_path_default?(base_url:)
-      host = URI.parse(base_url.to_s).host.to_s
-      ["localhost", "127.0.0.1"].include?(host)
-    rescue URI::InvalidURIError
-      false
-    end
   end
 end

data/lib/crawlscope/server_timing/reporter.rb

@@ -0,0 +1,112 @@
+# frozen_string_literal: true
+
+require "uri"
+
+module Crawlscope
+  class ServerTiming
+    class Reporter
+      def initialize(io:)
+        @io = io
+      end
+
+      def report(summary, base_url:)
+        return unless summary.present?
+
+        @io.puts("")
+        @io.puts("Server Timing:")
+        @io.puts(
+          "  Coverage: #{summary.coverage}/#{summary.pages.size} pages " \
+          "(#{percentage(summary.coverage, summary.pages.size)}); " \
+          "durations on #{summary.duration_pages} #{pluralize(:page, summary.duration_pages)}"
+        )
+
+        report_metrics(summary.metrics)
+        report_signals(summary.signals)
+        report_worst_pages(summary.worst_pages, base_url: base_url)
+        @io.puts("  Ignored malformed entries: #{summary.invalid_count}") if summary.invalid_count.positive?
+      end
+
+      private
+
+      def report_metrics(metrics)
+        return if metrics.empty?
+
+        @io.puts("  Duration metrics (dur; milliseconds assumed):")
+        metrics.each do |metric|
+          label = metric[:name]
+          label += " #{description(metric[:description])}" if metric[:description]
+
+          @io.puts(
+            "    #{label}: " \
+            "#{metric[:samples]} #{pluralize(:sample, metric[:samples])} / " \
+            "#{metric[:pages]} #{pluralize(:page, metric[:pages])}; " \
+            "avg #{duration(metric[:average])}; " \
+            "p50 #{duration(metric[:p50])}; " \
+            "p95 #{duration(metric[:p95])}; " \
+            "max #{duration(metric[:maximum])}"
+          )
+        end
+      end
+
+      def report_signals(signals)
+        return if signals.empty?
+
+        @io.puts("  Signals:")
+        signals.each do |signal|
+          label = signal[:name]
+          label += " #{description(signal[:description])}" if signal[:description]
+
+          @io.puts(
+            "    #{label}: " \
+            "#{signal[:samples]} #{pluralize(:sample, signal[:samples])} / " \
+            "#{signal[:pages]} #{pluralize(:page, signal[:pages])}"
+          )
+        end
+      end
+
+      def report_worst_pages(samples, base_url:)
+        return if samples.empty?
+
+        @io.puts("  Worst pages:")
+        samples.each do |sample|
+          metric = sample[:metric]
+          detail = metric.description ? " #{description(metric.description)}" : ""
+          @io.puts(
+            "    #{relative_url(sample[:page].url, base_url: base_url)}: " \
+            "#{duration(metric.duration)} (#{metric.name}#{detail})"
+          )
+        end
+      end
+
+      def duration(value)
+        "#{format("%.3f", value).sub(/\.?0+\z/, "")}ms"
+      end
+
+      def description(value)
+        value = value.to_s
+        value = "#{value[0, 77]}..." if value.length > 80
+        value.inspect
+      end
+
+      def percentage(numerator, denominator)
+        return "0.0%" if denominator.zero?
+
+        format("%.1f%%", numerator.fdiv(denominator) * 100)
+      end
+
+      def relative_url(url, base_url:)
+        uri = URI.parse(url)
+        base_uri = URI.parse(base_url)
+        return url unless uri.host == base_uri.host && uri.scheme == base_uri.scheme && uri.port == base_uri.port
+
+        uri.request_uri
+      rescue URI::InvalidURIError
+        url
+      end
+
+      def pluralize(word, count)
+        (count == 1) ? word.to_s : "#{word}s"
+      end
+    end
+  end
+end

data/lib/crawlscope/server_timing/summary.rb

@@ -0,0 +1,107 @@
+# frozen_string_literal: true
+
+module Crawlscope
+  class ServerTiming
+    class Summary
+      WORST_PAGE_LIMIT = 10
+
+      attr_reader :pages
+
+      def initialize(pages)
+        @pages = pages
+        @timings = pages.filter_map do |page|
+          timing = page.server_timing
+          [page, timing] if timing.present?
+        end
+      end
+
+      def present?
+        @timings.any?
+      end
+
+      def duration_pages
+        @timings.count { |_page, timing| timing.any?(&:timed?) }
+      end
+
+      def invalid_count
+        @timings.sum { |_page, timing| timing.invalid_count }
+      end
+
+      def metrics
+        timed_metrics
+          .group_by { |_page, metric| [metric.name, metric.description] }
+          .map { |(name, description), samples| metric_summary(name, description, samples) }
+          .sort_by { |metric| [-metric[:p95], -metric[:maximum], metric[:name]] }
+      end
+
+      def signals
+        all_metrics
+          .reject { |_page, metric| metric.timed? }
+          .group_by { |_page, metric| [metric.name, metric.description] }
+          .map { |(name, description), samples| signal_summary(name, description, samples) }
+          .sort_by { |signal| [-signal[:samples], signal[:name], signal[:description].to_s] }
+      end
+
+      def worst_pages(limit: WORST_PAGE_LIMIT)
+        @timings
+          .filter_map { |page, timing| worst_metric(page, timing) }
+          .min_by(limit) { |sample| [-sample[:metric].duration, sample[:page].url] }
+          .sort_by { |sample| [-sample[:metric].duration, sample[:page].url] }
+      end
+
+      def coverage
+        @timings.size
+      end
+
+      private
+
+      def timed_metrics
+        all_metrics.select { |_page, metric| metric.timed? }
+      end
+
+      def all_metrics
+        @all_metrics ||= @timings.flat_map do |page, timing|
+          timing.map { |metric| [page, metric] }
+        end
+      end
+
+      def metric_summary(name, description, samples)
+        durations = samples.map { |_page, metric| metric.duration }.sort
+
+        {
+          name: name,
+          description: description,
+          samples: samples.size,
+          pages: samples.map(&:first).uniq.size,
+          average: durations.sum.fdiv(durations.size),
+          p50: percentile(durations, 0.50),
+          p95: percentile(durations, 0.95),
+          maximum: durations.last
+        }
+      end
+
+      def signal_summary(name, description, samples)
+        {
+          name: name,
+          description: description,
+          samples: samples.size,
+          pages: samples.map(&:first).uniq.size
+        }
+      end
+
+      def percentile(values, percentile)
+        position = (values.size - 1) * percentile
+        lower = values[position.floor]
+        upper = values[position.ceil]
+
+        lower + (upper - lower) * (position - position.floor)
+      end
+
+      def worst_metric(page, timing)
+        if (metric = timing.select(&:timed?).max_by(&:duration))
+          {page: page, metric: metric}
+        end
+      end
+    end
+  end
+end