crawlscope 0.7.2 → 0.8.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 18717d5f025a5e0dc10b3f5a231b52d788a7ad09fe017b803815dc1cf22b7f11
-  data.tar.gz: 25f80cac563922d3c34d069c34b6c5521064fb4a74dd98cc54f6bb04e6de283a
+  metadata.gz: 89be661c49b5f2148a9d7962d81d522eedaedbba89c1f9dcf8474d62cfffd5f2
+  data.tar.gz: d27a26792cc5cfdab191371c3d005cf43298b02896cb7cbdf2113a5f341e6c12
 SHA512:
-  metadata.gz: 322dacf831bfa19dd16a6ebf3917e8d8bcaab462b440468945e4e9e07768a42955afe7f418b217594f9654c22490d32124f6fdb95004a20f23508f76bca6d7db
-  data.tar.gz: b301ad0f2c7c92dd3dc436f7019c62d7958bdd402eb8a887626d938a9bfd085d8d21bdab76a5243f3e2d9684d94ffce554e5108db26995d67b63140fcadbb9e2
+  metadata.gz: 5c90219d9430a09a3c95c5cb6367817cbc996f4ba69857781ddf6d2050373c9bfef2ad3de545a328542fa7362494df05be85cfa2c5e11f359b84870c8bee2ea6
+  data.tar.gz: 632ddff7f77d1c0eb62e6f9db0c9a36f8c3c080d2ac3b6b93bc03fcea65453d916ecbf7b39eb34e9b16d750309e7814586e2a84b4cd0157609a47402bcccf8ef

data/CHANGELOG.md

@@ -5,6 +5,33 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.8.1] - 2026-10-06
+
+
+### Fixed
+
+- clarify findings and prevent redirect false positives
+
+
+
+## [0.8.0] - 2026-07-24
+
+
+### Added
+
+- report Server-Timing metrics
+
+- authenticate timing crawl requests
+
+
+
+
+### Fixed
+
+- use portable crawl configuration defaults
+
+
+
 ## [0.7.2] - 2026-07-23
 
 

data/README.md

@@ -93,6 +93,11 @@ crawlscope validate --url https://example.com --sitemap https://example.com/site
 
 Child sitemap indexes are supported automatically.
 
+Set `CRAWLSCOPE_PROFILE_TOKEN` through the process environment or your secret
+manager when the site protects diagnostic timing with `X-Profile-Token`.
+Crawlscope intentionally has no `--profile-token` flag because command arguments
+can be exposed through shell history and process listings.
+
 Validation output is grouped for terminal scanning:
 
 ```text
@@ -101,21 +106,32 @@ Base URL: https://example.com
 Sitemap: https://example.com/sitemap.xml
 URLs: 24
 Pages: 24
-Status: FAILED
-Issues: 3 3 warnings
-
-Summary:
-  links            2
-  metadata         1
+Status: WARNINGS
+Issues: 3 total (3 warnings)
 
-links / low_dofollow_inlinks: 2
-  - /pricing  inbound 1/2  sources: /
-  - /features  inbound 1/2  sources: /
+Links: Too few incoming links without nofollow (2 warnings)
+  - /pricing  incoming links without nofollow: 1; minimum 2
+      linked from: /
+  - /features  incoming links without nofollow: 1; minimum 2
+      linked from: /
 
-metadata / missing_title: 1
+Metadata: Missing title (1 warning)
   - /draft  missing <title>
 ```
 
+Reports include all issues and related URLs. Errors appear before warnings and
+notices. Each related URL has a label, such as `linked from`, `links to`, or
+`page`. JSON-LD errors appear below the page URL with the type, field, and
+constraint. Same-origin URLs use paths; other URLs retain their scheme and host.
+For automation in Ruby, use `result.issues` and each issue's `code`, `severity`,
+`url`, and `details` rather than matching the report text.
+
+Metadata length limits appear in the group heading; each row shows the measured
+length. The default maximums are 160 characters for a meta description and 110
+characters for an Article JSON-LD `headline`. These are separate Crawlscope
+validation limits for separate fields. Server timing lists pages only when the
+reported duration exceeds 50 ms; aggregate statistics include all responses.
+
 ## Ruby Usage
 
 ```ruby
@@ -141,10 +157,69 @@ puts result.issues.to_a.map(&:message)
 - `urls`: sitemap URLs selected for validation
 - `pages`: fetched page snapshots
 - `issues`: structured issues with `code`, `severity`, `category`, `url`, and `message`
+- `server_timing_summary`: aggregate response timing data when pages publish it
 
 `result.ok?` returns `false` when an error is present. Warnings and notices
 remain available through `result.issues` without making the result fail.
 
+## Server Timing
+
+Crawlscope parses the
+[`Server-Timing`](https://www.w3.org/TR/server-timing/) response header for both
+HTTP and browser-rendered crawls. Each page exposes its parsed header through
+`page.server_timing`:
+
+```ruby
+page.server_timing.each do |metric|
+  puts [metric.name, metric.duration, metric.description].compact.join(": ")
+end
+```
+
+The validation report adds a `Server Timing` section only when at least one page
+publishes the header. It includes:
+
+- header coverage and the number of pages publishing durations
+- sample and page counts with average, p50, p95, and maximum per metric
+- non-duration signals, including cache status and routing descriptions
+- the ten pages with the largest individual metric
+- the number of malformed entries ignored during parsing
+
+`dur` values are reported as milliseconds, as recommended by the specification.
+Crawlscope does not add durations together because metrics such as `total`,
+`app`, and `db` may overlap. A page's worst-offender entry is its largest
+individual metric instead.
+
+The current HTTP and browser transports expose response headers, not response
+trailers. Metrics published only in a `Server-Timing` trailer are therefore not
+available to Crawlscope.
+
+Rails 8 applications can enable `ActionDispatch::ServerTiming` with:
+
+```ruby
+config.server_timing = true
+```
+
+Applications that expose timing only to authenticated diagnostics can set
+`config.profile_token`. When this value is present, Crawlscope sends it as
+`X-Profile-Token` on same-origin sitemap, HTTP, and browser document requests.
+The header survives same-origin redirects but is removed before a cross-origin
+redirect, and browser subresources never receive it. Keep the token in the host
+application's secret store rather than a URL, command argument, report, or
+checked-in configuration.
+
+`CRAWLSCOPE_PROFILE_TOKEN` is the portable default for Rails, the standalone
+CLI, Rake tasks, and plain Ruby callers. Rails applications may instead assign
+`config.profile_token` from encrypted credentials when that is their established
+secret-management boundary; explicit configuration takes precedence over the
+environment.
+
+New Rails applications enable it in development by default; production remains
+opt-in. Rails publishes the Active Support notification names observed during
+each request and sums repeated events with the same name. The exact metrics
+therefore depend on the application and request, but commonly include controller,
+view, database, and cache instrumentation. Crawlscope accepts Rails' dotted
+metric names and reports each metric independently.
+
 ## Rails Usage
 
 Run the install generator after adding the gem:
@@ -162,10 +237,11 @@ Customize the `Crawlscope.configure` block inside the generated initializer:
 
 ```ruby
 Crawlscope.configure do |config|
-  config.base_url = -> { "https://example.com" }
-  config.sitemap_path = -> { Rails.public_path.join("sitemap.xml").to_s }
-  config.site_name = "Example"
-  config.schema_registry = -> { Crawlscope::SchemaRegistry.default }
+  config.base_url = -> { ENV.fetch("CRAWLSCOPE_BASE_URL", "http://localhost:3000") }
+  config.sitemap_path = lambda {
+    ENV.fetch("SITEMAP", "#{config.base_url.to_s.chomp("/")}/sitemap.xml")
+  }
+  config.site_name = ENV.fetch("CRAWLSCOPE_SITE_NAME", "Application")
 end
 ```
 
@@ -193,6 +269,7 @@ Available environment overrides:
 
 - `URL`
 - `SITEMAP`
+- `CRAWLSCOPE_PROFILE_TOKEN`
 - `RULES=metadata,links`
 - `JS=1` or `RENDERER=browser`
 - `TIMEOUT=30`
@@ -226,10 +303,10 @@ bundle exec rake 'crawlscope:validate:ldjson[https://example.com/article]'
 
 `crawlscope:validate` runs all default sitemap rules: indexability, metadata,
 structured data, uniqueness, content quality, and links. `URL` is the site
-base. Without `SITEMAP`, Crawlscope uses the configured sitemap path, then
-falls back to `/sitemap.xml`. With `SITEMAP`, Crawlscope uses `URL` as the site
-base and validates URLs from that sitemap. `SITEMAP` may be a full URL or a
-local file path.
+base. Without `SITEMAP`, Crawlscope uses the configured sitemap URL, then fetches
+`/sitemap.xml` from `URL` over HTTP. With `SITEMAP`, Crawlscope uses `URL` as
+the site base and validates URLs from that sitemap. `SITEMAP` may be a full URL
+or an explicitly selected local file path.
 
 Plain `rake` does not pass `--url` style flags to tasks. Use `URL=...` or the
 task-argument form above instead.

data/lib/crawlscope/browser.rb

@@ -4,13 +4,15 @@ require "nokogiri"
 
 module Crawlscope
   class Browser
-    def initialize(base_url:, timeout_seconds:, network_idle_timeout_seconds:, scroll_page:)
+    def initialize(base_url:, timeout_seconds:, network_idle_timeout_seconds:, scroll_page:, profile_token: nil)
       @base_url = base_url
       @timeout_seconds = timeout_seconds
       @network_idle_timeout_seconds = network_idle_timeout_seconds
+      @profile_token = profile_token
       @scroll_page = scroll_page
       @browser = build_browser
       @page = @browser.create_page
+      configure_profile_requests
     end
 
     def close
@@ -72,6 +74,21 @@ module Crawlscope
       )
     end
 
+    def configure_profile_requests
+      return if @profile_token.to_s.empty?
+
+      @page.network.intercept(resource_type: :document)
+      @page.on(:request) do |request|
+        headers = RequestHeaders.add_profile_token(
+          request.headers.dup,
+          url: request.url,
+          base_url: @base_url,
+          profile_token: @profile_token
+        )
+        request.continue(headers: headers.map { |name, value| {name: name.to_s, value: value.to_s} })
+      end
+    end
+
     def scroll_for_render
       @page.evaluate("(function() { if (document.body) { window.scrollTo(0, document.body.scrollHeight); } })()")
       wait_for_network_idle

data/lib/crawlscope/configuration.rb

@@ -11,7 +11,7 @@ module Crawlscope
     RENDERERS = %i[http browser].freeze
     DEFAULT_TIMEOUT_SECONDS = 20
 
-    attr_writer :allowed_statuses, :base_url, :browser_factory, :concurrency, :fetch_executor, :network_idle_timeout_seconds, :output, :renderer, :rule_registry, :schema_registry, :scroll_page, :site_name, :sitemap_path, :timeout_seconds
+    attr_writer :allowed_statuses, :base_url, :browser_factory, :concurrency, :fetch_executor, :network_idle_timeout_seconds, :output, :profile_token, :renderer, :rule_registry, :schema_registry, :scroll_page, :site_name, :sitemap_path, :timeout_seconds
 
     def allowed_statuses
       value = resolve(@allowed_statuses)
@@ -59,6 +59,10 @@ module Crawlscope
       value.nil? ? $stdout : value
     end
 
+    def profile_token
+      resolve(@profile_token) || ENV["CRAWLSCOPE_PROFILE_TOKEN"]
+    end
+
     def renderer
       value = resolve(@renderer)
       normalized_value = value.to_s.strip
@@ -93,6 +97,7 @@ module Crawlscope
         concurrency: concurrency,
         fetch_executor: fetch_executor,
         network_idle_timeout_seconds: network_idle_timeout_seconds,
+        profile_token: profile_token,
         renderer: renderer,
         timeout_seconds: timeout_seconds,
         allowed_statuses: allowed_statuses,

data/lib/crawlscope/crawl.rb

@@ -2,7 +2,7 @@
 
 module Crawlscope
   class Crawl
-    def initialize(base_url:, sitemap_path:, rules:, schema_registry:, browser_factory: nil, concurrency: Configuration::DEFAULT_CONCURRENCY, fetch_executor: nil, network_idle_timeout_seconds: Configuration::DEFAULT_BROWSER_NETWORK_IDLE_TIMEOUT_SECONDS, renderer: :http, scroll_page: Configuration::DEFAULT_BROWSER_SCROLL_PAGE, timeout_seconds: Configuration::DEFAULT_TIMEOUT_SECONDS, allowed_statuses: Configuration::DEFAULT_ALLOWED_STATUSES)
+    def initialize(base_url:, sitemap_path:, rules:, schema_registry:, browser_factory: nil, concurrency: Configuration::DEFAULT_CONCURRENCY, fetch_executor: nil, network_idle_timeout_seconds: Configuration::DEFAULT_BROWSER_NETWORK_IDLE_TIMEOUT_SECONDS, profile_token: nil, renderer: :http, scroll_page: Configuration::DEFAULT_BROWSER_SCROLL_PAGE, timeout_seconds: Configuration::DEFAULT_TIMEOUT_SECONDS, allowed_statuses: Configuration::DEFAULT_ALLOWED_STATUSES)
       @base_url = base_url
       @sitemap_path = sitemap_path
       @rules = Array(rules)
@@ -10,6 +10,7 @@ module Crawlscope
       @browser_factory = browser_factory
       @concurrency = concurrency
       @network_idle_timeout_seconds = network_idle_timeout_seconds
+      @profile_token = profile_token
       @renderer = renderer.to_sym
       @fetch_executor = fetch_executor || default_fetch_executor
       @scroll_page = scroll_page
@@ -49,6 +50,7 @@ module Crawlscope
         adapter: http_adapter,
         concurrency: @concurrency,
         fetch_executor: @fetch_executor,
+        profile_token: @profile_token,
         timeout_seconds: @timeout_seconds
       ).urls(base_url: @base_url)
       raise ValidationError, "No URLs found in sitemap: #{@sitemap_path}" if urls.empty?
@@ -61,6 +63,7 @@ module Crawlscope
         base_url: @base_url,
         timeout_seconds: @timeout_seconds,
         network_idle_timeout_seconds: @network_idle_timeout_seconds,
+        profile_token: @profile_token,
         scroll_page: @scroll_page
       )
     rescue LoadError => error
@@ -71,7 +74,7 @@ module Crawlscope
       if @renderer == :browser
         (@browser_factory || method(:browser)).call
       else
-        Http.new(base_url: @base_url, timeout_seconds: @timeout_seconds, adapter: http_adapter)
+        Http.new(base_url: @base_url, timeout_seconds: @timeout_seconds, adapter: http_adapter, profile_token: @profile_token)
       end
     end
 
@@ -111,7 +114,6 @@ module Crawlscope
         elsif !@allowed_statuses.include?(page.status)
           issues.add(code: :unexpected_status, severity: :error, category: :crawl, url: page.url, message: "HTTP #{page.status}", details: {status: page.status})
         elsif redirected?(page)
-          issues.add(code: :redirected_page, severity: :warning, category: :crawl, url: page.url, message: "redirects to #{page.final_url}", details: {final_url: page.final_url, status: page.status})
           issues.add(code: :sitemap_redirect_url, severity: :warning, category: :sitemaps, url: page.url, message: "sitemap URL redirects to #{page.final_url}", details: {final_url: page.final_url, status: page.status})
         end
       end

data/lib/crawlscope/http.rb

@@ -10,10 +10,11 @@ module Crawlscope
     MAX_REDIRECTS = 5
     USER_AGENT = "Mozilla/5.0 (compatible; Crawlscope/1.0)"
 
-    def initialize(base_url:, timeout_seconds:, adapter: nil)
+    def initialize(base_url:, timeout_seconds:, adapter: nil, profile_token: nil)
       @base_url = base_url
       @timeout_seconds = timeout_seconds
       @adapter = adapter
+      @profile_token = profile_token
       @connections_by_thread = Concurrent::Map.new
     end
 
@@ -28,6 +29,12 @@ module Crawlscope
     def fetch(url)
       response = connection.get(url) do |request|
         request.headers["User-Agent"] = USER_AGENT
+        RequestHeaders.add_profile_token(
+          request.headers,
+          url: url,
+          base_url: @base_url,
+          profile_token: @profile_token
+        )
       end
 
       final_url = response.env.url.to_s
@@ -67,7 +74,11 @@ module Crawlscope
     def connection
       @connections_by_thread.compute_if_absent(Thread.current.object_id) do
         Faraday.new do |faraday|
-          faraday.response :follow_redirects, limit: MAX_REDIRECTS
+          faraday.response(
+            :follow_redirects,
+            limit: MAX_REDIRECTS,
+            callback: RequestHeaders.method(:strip_profile_token_on_cross_origin_redirect)
+          )
           faraday.options.timeout = @timeout_seconds
           faraday.options.open_timeout = @timeout_seconds
           faraday.adapter @adapter if @adapter

data/lib/crawlscope/page.rb

@@ -19,5 +19,13 @@ module Crawlscope
     def html?
       !doc.nil?
     end
+
+    def header(name)
+      headers.find { |key, _value| key.to_s.casecmp?(name.to_s) }&.last
+    end
+
+    def server_timing
+      @server_timing ||= ServerTiming.new(header("server-timing"))
+    end
   end
 end

data/lib/crawlscope/reporter.rb

@@ -4,7 +4,37 @@ require "uri"
 
 module Crawlscope
   class Reporter
-    MAX_ISSUES_PER_GROUP = 20
+    GROUP_LABELS = {
+      structured_data_schema_error: "Schema errors",
+      structured_data_parse_error: "Parse errors",
+      structured_data_missing_type: "Missing @type",
+      sitemap_redirect_url: "Redirected URLs",
+      non_canonical_page_in_sitemap: "URLs with a different canonical",
+      indexable_page_missing_from_sitemap: "Linked pages missing from sitemap",
+      sitemap_noindex_url: "URLs that block indexing",
+      low_dofollow_inlinks: "Too few incoming links without nofollow",
+      low_inbound_anchor_links: "Too few incoming links",
+      low_unique_token_ratio: "Repeated words",
+      low_visible_text_ratio: "Too little text for the HTML size",
+      thin_visible_text: "Too few words",
+      orphan_page: "Pages with no incoming links",
+      nofollow_internal_outlinks: "Outgoing links with nofollow",
+      only_nofollow_internal_inlinks: "Incoming links all use nofollow",
+      mixed_follow_internal_inlinks: "Incoming links with and without nofollow",
+      http_internal_link: "Links from HTTPS to HTTP",
+      canonical_no_internal_inlinks: "Canonical URLs with no incoming links without nofollow",
+      canonical_points_to_error: "Canonical URLs that return errors",
+      url_double_slash: "Repeated slashes in URL paths",
+      multiple_h1: "Multiple <h1> tags",
+      multiple_title_tags: "Multiple <title> tags",
+      incomplete_open_graph_tags: "Missing Open Graph tags",
+      title_too_long: "Title exceeds #{Rules::Metadata::TITLE_MAX_LENGTH} characters",
+      meta_description_too_long: "Meta description exceeds #{Rules::Metadata::DESCRIPTION_MAX_LENGTH} characters",
+      meta_description_too_short: "Meta description below #{Rules::Metadata::DESCRIPTION_MIN_LENGTH} characters",
+      duplicate_content_fingerprint: "Duplicate page text",
+      near_duplicate_content: "Similar page text",
+      near_duplicate_scan_skipped: "Similar-page check skipped"
+    }.freeze
 
     def initialize(io:)
       @io = io
@@ -19,16 +49,17 @@ module Crawlscope
 
       if result.issues.size.zero?
         @io.puts("Status: OK")
-        return
+      else
+        @io.puts("Status: #{status_for(result.issues)}")
+        @io.puts("Issues: #{result.issues.size} total (#{severity_summary(result.issues)})")
       end
 
-      @io.puts("Status: #{status_for(result.issues)}")
-      @io.puts("Issues: #{result.issues.size} total (#{severity_summary(result.issues)})")
-      @io.puts("")
+      unless result.issues.size.zero?
+        @io.puts("")
+        report_issue_groups(result.issues, base_url: result.base_url)
+      end
 
-      report_summary(result.issues)
-      @io.puts("")
-      report_issue_groups(result.issues, base_url: result.base_url)
+      ServerTiming::Reporter.new(io: @io).report(result.server_timing_summary, base_url: result.base_url)
     end
 
     private
@@ -46,43 +77,45 @@ module Crawlscope
     end
 
     def severity_summary(issues)
-      grouped = issues.by_severity
-      return "" if grouped.empty?
-
-      grouped
-        .sort_by { |severity, severity_issues| [-severity_issues.size, severity.to_s] }
-        .map { |severity, severity_issues| "#{severity_issues.size} #{pluralize(severity, severity_issues.size)}" }
-        .join(", ")
-    end
-
-    def report_summary(issues)
-      @io.puts("Summary:")
-
-      issues.by_category
-        .sort_by { |category, category_issues| [-category_issues.size, category.to_s] }
-        .each do |category, category_issues|
-          @io.puts("  #{category.to_s.ljust(16)} #{category_issues.size}")
-        end
+      severity_summary_for(issues.to_a)
     end
 
     def report_issue_groups(issues, base_url:)
       grouped = issues.to_a.group_by { |issue| [issue.category, issue.code] }
 
       grouped
-        .sort_by { |(category, code), grouped_issues| [-grouped_issues.size, category.to_s, code.to_s] }
+        .sort_by { |(category, code), grouped_issues| [grouped_issues.map { |issue| severity_order(issue.severity) }.min, -grouped_issues.size, category.to_s, code.to_s] }
         .each do |(category, code), grouped_issues|
-          @io.puts("#{category} / #{code}: #{grouped_issues.size}")
-
-          grouped_issues.first(MAX_ISSUES_PER_GROUP).each do |issue|
-            @io.puts("  - #{compact_issue(issue, base_url: base_url)}")
+          label = GROUP_LABELS.fetch(code) { code.to_s.tr("_", " ").capitalize }
+          @io.puts("#{category.to_s.tr("_", " ").capitalize}: #{label} (#{severity_summary_for(grouped_issues)})")
+
+          grouped_issues.each do |issue|
+            details = issue.details || {}
+            if issue.code == :structured_data_schema_error && details[:errors]&.any?
+              source = (details[:source] == "json-ld") ? "JSON-LD" : details[:source]
+              @io.puts("  - #{relative_url(issue.url, base_url: base_url)}  #{source}")
+              details[:errors].each do |error|
+                @io.puts("      #{StructuredData::ErrorFormatter.format(error)}")
+              end
+            else
+              @io.puts("  - #{compact_issue(issue, base_url: base_url)}")
+              report_related_urls(issue, details, base_url: base_url)
+            end
           end
 
-          remaining_count = grouped_issues.size - MAX_ISSUES_PER_GROUP
-          @io.puts("  ... #{remaining_count} more") if remaining_count.positive?
           @io.puts("")
         end
     end
 
+    def severity_order(severity)
+      {error: 0, warning: 1, notice: 2}.fetch(severity, 3)
+    end
+
+    def severity_summary_for(issues)
+      issues.group_by(&:severity).sort_by { |severity, _| severity_order(severity) }
+        .map { |severity, entries| "#{entries.size} #{pluralize(severity, entries.size)}" }.join(", ")
+    end
+
     def compact_issue(issue, base_url:)
       parts = []
       parts << relative_url(issue.url, base_url: base_url) if issue.url
@@ -95,35 +128,87 @@ module Crawlscope
 
     def compact_detail(issue, base_url:)
       details = issue.details || {}
-      fragments = []
-
-      inbound = details[:dofollow_inbound_count] || details[:inbound_count]
-      fragments << "inbound #{inbound}/#{details[:minimum]}" if inbound && details[:minimum]
-
-      if details[:ratio] && details[:threshold]
-        fragments << "ratio #{format_number(details[:ratio])}/#{format_number(details[:threshold])}"
-      end
-
-      fragments << "count #{details[:count]}" if details[:count]
-      fragments << "length #{details[:length]}" if details[:length]
-      fragments << "status #{details[:status]}" if details[:status]
-      fragments << "final: #{relative_url(details[:final_url], base_url: base_url)}" if details[:final_url]
-      fragments << "sources: #{relative_urls(details[:source_urls], base_url: base_url).join(", ")}" if details[:source_urls]&.any?
-      fragments << "source: #{relative_url(details[:source_url], base_url: base_url)}" if details[:source_url]
-      fragments << "targets: #{relative_urls(details[:target_urls], base_url: base_url).join(", ")}" if details[:target_urls]&.any?
-
-      return issue.message if fragments.empty?
-
       case issue.code
-      when :low_dofollow_inlinks, :low_inbound_anchor_links, :low_unique_token_ratio, :low_visible_text_ratio
-        fragments.join("  ")
-      else
-        ([issue.message] + fragments).join("  ")
+      when :redirected_page, :sitemap_redirect_url, :internal_link_redirects
+        if details[:final_url]
+          prefix = (issue.code == :sitemap_redirect_url) ? "sitemap URL redirects to" : "redirects to"
+          status = details[:status] ? " (final HTTP #{details[:status]})" : ""
+          return "#{prefix} #{relative_url(details[:final_url], base_url: base_url)}#{status}"
+        end
+      when :canonical_mismatch, :non_canonical_page_in_sitemap
+        if details[:canonical]
+          prefix = (issue.code == :non_canonical_page_in_sitemap) ? "sitemap lists this URL, but its canonical points to" : "canonical points to a different URL:"
+          return "#{prefix} #{relative_url(details[:canonical], base_url: base_url)}"
+        end
+      when :unexpected_status
+        return "HTTP #{details[:status]}" if details[:status]
+      when :title_too_long, :meta_description_too_long, :meta_description_too_short
+        return "#{details[:length]} characters" if details[:length]
+      when :broken_internal_link
+        return "linked URL returns HTTP #{details[:status]}"
+      when :unresolved_internal_link
+        return ["cannot check linked URL", details[:error]].compact.join(": ")
+      when :indexable_page_missing_from_sitemap
+        return "linked page is missing from sitemap"
+      when :canonical_points_to_redirect
+        status = details[:status] ? " (final HTTP #{details[:status]})" : ""
+        return "canonical #{relative_url(details[:canonical], base_url: base_url)} redirects to #{relative_url(details[:final_url], base_url: base_url)}#{status}"
+      when :canonical_points_to_error
+        return "canonical #{relative_url(details[:canonical], base_url: base_url)} returns HTTP #{details[:status]}"
+      when :low_dofollow_inlinks, :low_inbound_anchor_links
+        count = details[:dofollow_inbound_count] || details[:inbound_count]
+        label = (issue.code == :low_dofollow_inlinks) ? "incoming links without nofollow" : "incoming links"
+        return "#{label}: #{count}; minimum #{details[:minimum]}"
+      when :low_unique_token_ratio, :low_visible_text_ratio
+        label = (issue.code == :low_unique_token_ratio) ? "unique words" : "visible text / HTML bytes"
+        return "#{label}: #{percentage(details[:ratio])}; minimum #{percentage(details[:threshold])}"
+      when :thin_visible_text
+        return "visible words: #{details[:word_count]}; minimum #{details[:minimum]}"
+      when :mixed_follow_internal_inlinks
+        return "incoming links: #{details[:dofollow_inbound_count]} without nofollow, #{details[:nofollow_inbound_count]} with nofollow"
+      when :only_nofollow_internal_inlinks
+        return "all #{details[:nofollow_inbound_count]} incoming internal links use nofollow"
+      when :duplicate_title, :duplicate_meta_description
+        label = (issue.code == :duplicate_title) ? "same title" : "same meta description"
+        return "#{label}: #{details[:value].to_s.inspect}"
+      when :duplicate_content_fingerprint
+        return "same page text"
+      when :duplicate_pages_without_canonical
+        return "same page text; at least one page has no canonical link"
+      when :near_duplicate_content
+        return "text similarity: #{percentage(details[:similarity])}; threshold #{percentage(details[:threshold])}"
+      when :near_duplicate_scan_skipped
+        return "similar-page check skipped: #{details[:page_count]} pages; limit #{details[:max_pages]}"
+      when :structured_data_missing_type
+        return "#{issue.message} at #{Array(details[:paths]).join(", ")}" if details[:paths]&.any?
+      when :noindex_meta, :nofollow_meta, :noindex_follow_meta, :noindex_nofollow_meta,
+          :noindex_header, :nofollow_header, :noindex_follow_header, :noindex_nofollow_header,
+          :sitemap_noindex_url
+        if details[:content]
+          source = details[:name] || details[:source]
+          label = source ? "#{source}: " : ""
+          return "#{issue.message} (#{label}#{details[:content]})"
+        end
       end
+      issue.message
     end
 
-    def relative_urls(urls, base_url:)
-      Array(urls).map { |url| relative_url(url, base_url: base_url) }
+    def report_related_urls(issue, details, base_url:)
+      source_label = (issue.code == :canonical_no_internal_inlinks) ? "canonical on" : "linked from"
+      sources = Array(details[:source_urls])
+      sources += [details[:source_url]] if details[:source_url]
+      relationships = {
+        source_label => sources.uniq,
+        "links to" => details[:target_urls],
+        "page" => details[:urls],
+        "linked from (without nofollow)" => details[:dofollow_source_urls],
+        "linked from (nofollow)" => details[:nofollow_source_urls]
+      }
+      relationships.each do |label, urls|
+        Array(urls).each do |url|
+          @io.puts("      #{label}: #{relative_url(url, base_url: base_url)}")
+        end
+      end
     end
 
     def relative_url(url, base_url:)
@@ -142,10 +227,8 @@ module Crawlscope
       url
     end
 
-    def format_number(value)
-      return format("%.3f", value) if value.is_a?(Float)
-
-      value.to_s
+    def percentage(value)
+      "#{format("%.3f", value * 100).sub(/0+\z/, "").sub(/\.\z/, "")}%"
     end
 
     def pluralize(word, count)