crawlscope 0.10.0 → 0.10.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: da58b612eb199f881ec2263858feea4bf93e11355ffce7b19921bb266451818e
-  data.tar.gz: bcfa91a8bd74cf5ae670dcc5a1b3e762ad0c6a74037cd82f92b700f8b3a2d7d0
+  metadata.gz: 1ff240a395648f3047e4fd24fe2728ad5c7ad3c22406029f7dbb0316f43bee73
+  data.tar.gz: 93e012e28037929afadba5df00f8ad441665ddda8cbc55775278fcc427d5af00
 SHA512:
-  metadata.gz: e3001de98212ddd2dca5f15feec4bf32eb9b3a975bef21c948c15c588c9a6d9f18bd130985fc3a6cfa78d1dfd34c2c5c4e944563ecd27be87cc49819475aed46
-  data.tar.gz: 784d47ba0dbf3effdbacb53d9211472e044b22da28c2d00e21bbcb8844aa2a1309eba072c0c631412b1dd7bad10f6713ad0adb107f2805df42310b1b31ad7991
+  metadata.gz: d7f51921917c37fd1cb6576d1c1c56957cdc6be141906b5958153c548296248a76316832efd7dcd562835a08b6393a3009e577ed59f676b72786ec3dc99cd81b
+  data.tar.gz: 7db8cd080f1256e42e17cf09cbd3a34f2a62bcbd0e48238e58f420359519fd4e737516c21ad48212c17bc34e093914e641d2a8fbbd60ec94f68600466ae098bf

data/CHANGELOG.md

@@ -5,6 +5,17 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.10.1] - 2026-11-08
+
+
+### Documentation
+
+- make Ethos Link the canonical documentation home
+
+- keep audited technical guidance alongside the code
+
+
+
 ## [0.10.0] - 2026-11-06
 
 

data/README.md

@@ -3,13 +3,13 @@
 [![Gem Version](https://badge.fury.io/rb/crawlscope.svg)](https://badge.fury.io/rb/crawlscope)
 [![Commit messages](https://github.com/ethos-link/crawlscope/actions/workflows/commitlint.yml/badge.svg)](https://github.com/ethos-link/crawlscope/actions/workflows/commitlint.yml)
 
-`crawlscope` is a small Ruby gem for sitemap-driven SEO validation.
+`crawlscope` audits sitemap URLs for indexability, metadata, structured data and internal links.
 
 It is built by [Ethos Link](https://www.ethos-link.com) and used in production by [Reviato](https://www.reviato.com).
 
 It is designed for Rails apps and plain Ruby scripts that want:
 
-- deterministic sitemap crawling
+- sitemap-driven crawling
 - structured validation issues instead of free-form strings
 - app-configurable rule and schema registries
 - first-party rake tasks instead of a large DSL
@@ -30,6 +30,16 @@ The default rule set includes:
 - content-quality checks
 - internal-link checks
 
+## Documentation
+
+This README and the linked repository guides describe the checked-out code.
+The [Ethos Link project page](https://www.ethos-link.com/opensource/crawlscope)
+provides the canonical public project guide, overview and resources.
+
+- [Rails integration and tasks](docs/rails.md)
+- [Rule catalogue and schema registries](docs/rules.md)
+- [Server timing and diagnostic headers](docs/server-timing.md)
+
 ## Installation
 
 Crawlscope requires Ruby 4.0 or newer. Rails is optional; the CLI and Ruby
@@ -63,7 +73,7 @@ If you want browser rendering, also add:
 gem "ferrum"
 ```
 
-`crawlscope` only loads Ferrum when you run in browser mode.
+`crawlscope` only loads Ferrum when you run in browser mode. Install a compatible Chrome or Chromium executable as well.
 
 ## CLI Usage
 
@@ -163,277 +173,46 @@ puts result.issues.to_a.map(&:message)
 `result.ok?` returns `false` when an error is present. Warnings and notices
 remain available through `result.issues` without making the result fail.
 
-## Server Timing
-
-Crawlscope parses the
-[`Server-Timing`](https://www.w3.org/TR/server-timing/) response header for both
-HTTP and browser-rendered crawls. Each page exposes its parsed header through
-`page.server_timing`:
-
-```ruby
-page.server_timing.each do |metric|
-  puts [metric.name, metric.duration, metric.description].compact.join(": ")
-end
-```
-
-The validation report adds a `Server Timing` section only when at least one page
-publishes the header. It includes:
-
-- header coverage and the number of pages publishing durations
-- sample and page counts with average, p50, p95, and maximum per metric
-- non-duration signals, including cache status and routing descriptions
-- the ten pages with the largest individual metric
-- the number of malformed entries ignored during parsing
-
-`dur` values are reported as milliseconds, as recommended by the specification.
-Crawlscope does not add durations together because metrics such as `total`,
-`app`, and `db` may overlap. A page's worst-offender entry is its largest
-individual metric instead.
-
-The current HTTP and browser transports expose response headers, not response
-trailers. Metrics published only in a `Server-Timing` trailer are therefore not
-available to Crawlscope.
+## Rails integration
 
-Rails 8 applications can enable `ActionDispatch::ServerTiming` with:
+Add `gem "crawlscope", require: false` to the host application's Gemfile, run
+`bundle install`, then install the task/configuration integration:
 
-```ruby
-config.server_timing = true
-```
-
-Applications that expose timing only to authenticated diagnostics can set
-`config.profile_token`. When this value is present, Crawlscope sends it as
-`X-Profile-Token` on same-origin sitemap, HTTP, and browser document requests.
-The header survives same-origin redirects but is removed before a cross-origin
-redirect, and browser subresources never receive it. Keep the token in the host
-application's secret store rather than a URL, command argument, report, or
-checked-in configuration.
-
-`CRAWLSCOPE_PROFILE_TOKEN` is the portable default for Rails, the standalone
-CLI, Rake tasks, and plain Ruby callers. Rails applications may instead assign
-`config.profile_token` from encrypted credentials when that is their established
-secret-management boundary; explicit configuration takes precedence over the
-environment.
-
-New Rails applications enable it in development by default; production remains
-opt-in. Rails publishes the Active Support notification names observed during
-each request and sums repeated events with the same name. The exact metrics
-therefore depend on the application and request, but commonly include controller,
-view, database, and cache instrumentation. Crawlscope accepts Rails' dotted
-metric names and reports each metric independently.
-
-## Rails Usage
-
-Install Rails 8 in the host application; Crawlscope does not install Railties.
-For a process that loads Crawlscope before Rails, require `crawlscope/rails`
-after Rails is available. The Ruby and CLI entry points remain independent of Rails.
-
-Run the install generator after adding the gem:
-
-```bash
+```sh
 bin/rails generate crawlscope:install
+bin/rails crawlscope:validate
 ```
 
-The generator creates `config/initializers/crawlscope.rb` with an idempotent
-`CrawlscopeConfiguration.apply` loader and adds conditional `crawlscope/tasks`
-loading to the application's `Rakefile`. This preserves `bin/rails --tasks` and
-the `crawlscope:*` tasks without loading Crawlscope during normal Rails boot.
-
-Customize the `Crawlscope.configure` block inside the generated initializer:
-
-```ruby
-Crawlscope.configure do |config|
-  config.base_url = -> { ENV.fetch("CRAWLSCOPE_BASE_URL", "http://localhost:3000") }
-  config.sitemap_path = lambda {
-    ENV.fetch("SITEMAP", "#{config.base_url.to_s.chomp("/")}/sitemap.xml")
-  }
-  config.site_name = ENV.fetch("CRAWLSCOPE_SITE_NAME", "Application")
-end
-```
-
-Runtime callers must apply the generated configuration before using
-Crawlscope:
+Edit the generated `Crawlscope.configure` block to set `base_url`,
+`sitemap_path` and any custom rule/schema registries. Tasks load and apply it
+automatically. From runtime application code, apply the generated loader and
+execute the crawl explicitly:
 
 ```ruby
 CrawlscopeConfiguration.apply
-Crawlscope.configuration.audit
-```
-
-Rake tasks apply it automatically because `crawlscope/tasks` loads the gem
-before Rails evaluates the initializer. Each Rake entry point passes the shared
-`Crawlscope.configuration` object to the CLI, so configured base URLs, sitemap
-paths, registries, and runtime settings are preserved unless a task argument or
-environment override replaces them.
-
-Then run:
-
-```bash
-bin/rails crawlscope:validate
-```
-
-Available environment overrides:
-
-- `URL`
-- `SITEMAP`
-- `CRAWLSCOPE_PROFILE_TOKEN`
-- `RULES=metadata,links`
-- `JS=1` or `RENDERER=browser`
-- `TIMEOUT=30`
-- `NETWORK_IDLE_TIMEOUT=10`
-- `CONCURRENCY=5`
-- `FETCH_EXECUTOR=threaded` or `FETCH_EXECUTOR=async`
-
-Available tasks:
-
-```bash
-bin/rails crawlscope:validate
-bin/rails crawlscope:validate:indexability
-bin/rails crawlscope:validate:metadata
-bin/rails crawlscope:validate:structured_data
-bin/rails crawlscope:validate:uniqueness
-bin/rails crawlscope:validate:content_quality
-bin/rails crawlscope:validate:links
-bin/rails crawlscope:validate:ldjson
-```
-
-The same validation surface is also available in the gem repository itself through plain `rake`:
-
-```bash
-bundle exec rake crawlscope:validate URL=https://example.com
-bundle exec rake 'crawlscope:validate[https://example.com]'
-bundle exec rake crawlscope:validate:metadata URL=https://example.com
-bundle exec rake 'crawlscope:validate:metadata[https://example.com]'
-bundle exec rake crawlscope:validate:ldjson URL=https://example.com/article
-bundle exec rake 'crawlscope:validate:ldjson[https://example.com/article]'
-```
-
-`crawlscope:validate` runs all default sitemap rules: indexability, metadata,
-structured data, uniqueness, content quality, and links. `URL` is the site
-base. Without `SITEMAP`, Crawlscope uses the configured sitemap URL, then fetches
-`/sitemap.xml` from `URL` over HTTP. With `SITEMAP`, Crawlscope uses `URL` as
-the site base and validates URLs from that sitemap. `SITEMAP` may be a full URL
-or an explicitly selected local file path.
-
-Plain `rake` does not pass `--url` style flags to tasks. Use `URL=...` or the
-task-argument form above instead.
-
-`FETCH_EXECUTOR=async` is the default for HTTP crawling. It uses Ruby's fiber
-scheduler and Async::HTTP through Faraday, preserving the same `CONCURRENCY`
-bound. Use `FETCH_EXECUTOR=threaded` or `--fetch-executor threaded` for the
-thread-pool executor. Browser rendering uses the threaded executor by default
-because async fetch execution is only supported with HTTP rendering.
-
-`crawlscope:validate:ldjson` is separate because it directly checks the URL or semicolon-separated URLs in `URL`; it does not crawl the sitemap. Without `URL`, it checks the configured base URL, falling back to `http://localhost:3000`.
-
-### Structured Data URL Audit
-
-For one-off structured-data checks:
-
-```bash
-bin/rails crawlscope:validate:ldjson URL=https://example.com/article
-bin/rails crawlscope:validate:ldjson URL='https://example.com/a;https://example.com/b' SUMMARY=1
-bin/rails crawlscope:validate:ldjson URL=https://example.com/article REPORT_PATH=tmp/structured-data.json
+result = Crawlscope.configuration.audit.call
 ```
 
-Optional flags:
-
-- `DEBUG=1`: print detected items
-- `SUMMARY=1`: print grouped failures
-- `REPORT_PATH=...`: write a JSON report. Treat this as trusted operator input; Crawlscope writes to the path the task process can access.
-- `JS=1` or `RENDERER=browser`: render with Ferrum
-
-## Rules
-
-Built-in rules:
-
-- `indexability`
-- `metadata`
-- `structured_data`
-- `uniqueness`
-- `content_quality`
-- `links`
-
-### Indexability
-
-Checks:
-
-- page-level meta robots `noindex`
-- `X-Robots-Tag: noindex`
-
-### Metadata
-
-Checks:
-
-- missing `<h1>`
-- missing `<title>`
-- title length
-- repeated site name in the title
-- missing meta description
-- meta description length
-- missing canonical link
-- canonical mismatch
-
-### Structured Data
+See [Rails integration](docs/rails.md) for lazy loading, all tasks,
+environment overrides and structured-data URL audits.
 
-Checks:
+## Configuration and limits
 
-- malformed JSON-LD
-- missing required fields for supported schema types
-- schema validation failures from the configured registry
-- direct URL structured-data audits through `crawlscope:validate:ldjson`
+The shared configuration defaults to HTTP rendering, the async fetch executor,
+10 concurrent requests and a 20-second request timeout. Browser rendering uses
+the threaded executor with at most four concurrent browser fetches. Configure
+`concurrency`, `timeout_seconds`, `renderer`, `fetch_executor`, `rule_registry`
+and `schema_registry` through `Crawlscope.configure` when using that shared
+configuration; direct `Crawlscope::Crawl.new` callers pass their own options.
 
-### Uniqueness
+Use [rule and schema registries](docs/rules.md) to select or extend checks.
+Near-duplicate scanning is limited to 250 HTML pages by default; exact duplicate
+checks still run for larger crawls. Validation describes observed responses and
+configured rules; it does not establish Google rich-result eligibility or
+search ranking.
 
-Checks:
-
-- duplicate titles
-- duplicate meta descriptions
-- duplicate content fingerprints
-- near-duplicate visible content for up to 250 HTML pages
-
-For larger crawls, exact duplicate checks still run and Crawlscope reports
-`near_duplicate_scan_skipped`. Configure `Rules::Uniqueness` with
-`max_near_duplicate_pages:` in a custom rule registry to change the limit.
-
-### Content Quality
-
-Checks:
-
-- thin visible text
-- low visible-text-to-HTML ratio
-- low unique-token ratio
-
-### Links
-
-Checks:
-
-- broken internal links
-- unresolved internal links
-- low inbound anchor-link counts
-
-## Schema Registry
-
-`crawlscope` ships with a default schema registry for common types such as:
-
-- `Article`
-- `FAQPage`
-- `Organization`
-- `Product`
-- `Review`
-- `SoftwareApplication`
-- `WebApplication`
-- `WebSite`
-
-The default schema definitions live in `Crawlscope::Schemas`; `Crawlscope::SchemaRegistry` owns registration and validation.
-
-Host apps can replace or extend the registry:
-
-```ruby
-Crawlscope.configure do |config|
-  config.schema_registry = -> { MyApp::StructuredData::SchemaRegistry.new }
-end
-```
-
-That makes `crawlscope` useful as the audit engine while the app remains the owner of stricter product-specific schema rules.
+[Server timing](docs/server-timing.md) covers response timings, diagnostic
+profile tokens and the limits of those measurements.
 
 ## Contributing
 
@@ -463,3 +242,15 @@ bundle exec rake verify
 Tests use controlled local fixtures and need no service account.
 Release preparation, publication, registry setup and recovery are documented
 in the [Gemkit release guide](https://github.com/ethos-link/ethos-gemkit#releasing-a-gem).
+
+## About Ethos Link
+
+[Ethos Link](https://www.ethos-link.com) is a software company founded by
+[Paulo Fidalgo](https://www.fidalgo.pt). This gem is part of the open-source
+tooling we publish alongside our business products:
+
+- [Reviato](https://www.reviato.com): collect private customer feedback and analyse public reviews.
+- [Nuntio](https://www.nuntiohq.com): prepare, review and schedule social posts.
+- [EthosPlaces](https://www.ethos.place): find places, opening hours and contact details.
+
+Explore our [open-source projects](https://www.ethos-link.com/opensource).

data/docs/rails.md

@@ -0,0 +1,121 @@
+# Rails integration
+
+[README](../README.md) · [Project guide](https://www.ethos-link.com/opensource/crawlscope)
+
+Install Rails 8 in the host application; Crawlscope does not install Railties.
+For a process that loads Crawlscope before Rails, require `crawlscope/rails`
+after Rails is available. The Ruby and CLI entry points remain independent of Rails.
+
+Run the install generator after adding the gem:
+
+```bash
+bin/rails generate crawlscope:install
+```
+
+The generator creates `config/initializers/crawlscope.rb` with an idempotent
+`CrawlscopeConfiguration.apply` loader and adds conditional `crawlscope/tasks`
+loading to the application's `Rakefile`. This preserves `bin/rails --tasks` and
+the `crawlscope:*` tasks without loading Crawlscope during normal Rails boot.
+
+Customize the `Crawlscope.configure` block inside the generated initializer:
+
+```ruby
+Crawlscope.configure do |config|
+  config.base_url = -> { ENV.fetch("CRAWLSCOPE_BASE_URL", "http://localhost:3000") }
+  config.sitemap_path = lambda {
+    ENV.fetch("SITEMAP", "#{config.base_url.to_s.chomp("/")}/sitemap.xml")
+  }
+  config.site_name = ENV.fetch("CRAWLSCOPE_SITE_NAME", "Application")
+end
+```
+
+Runtime callers must apply the generated configuration before using
+Crawlscope:
+
+```ruby
+CrawlscopeConfiguration.apply
+Crawlscope.configuration.audit.call
+```
+
+Rake tasks apply it automatically because `crawlscope/tasks` loads the gem
+before Rails evaluates the initializer. Each Rake entry point passes the shared
+`Crawlscope.configuration` object to the CLI, so configured base URLs, sitemap
+paths, registries, and runtime settings are preserved unless a task argument or
+environment override replaces them.
+
+Then run:
+
+```bash
+bin/rails crawlscope:validate
+```
+
+Available environment overrides:
+
+- `URL`
+- `SITEMAP`
+- `CRAWLSCOPE_PROFILE_TOKEN`
+- `RULES=metadata,links`
+- `JS=1` or `RENDERER=browser`
+- `TIMEOUT=30`
+- `NETWORK_IDLE_TIMEOUT=10`
+- `CONCURRENCY=5`
+- `FETCH_EXECUTOR=threaded` or `FETCH_EXECUTOR=async`
+
+Available tasks:
+
+```bash
+bin/rails crawlscope:validate
+bin/rails crawlscope:validate:indexability
+bin/rails crawlscope:validate:metadata
+bin/rails crawlscope:validate:structured_data
+bin/rails crawlscope:validate:uniqueness
+bin/rails crawlscope:validate:content_quality
+bin/rails crawlscope:validate:links
+bin/rails crawlscope:validate:ldjson
+```
+
+The same validation surface is also available in the gem repository itself through plain `rake`:
+
+```bash
+bundle exec rake crawlscope:validate URL=https://example.com
+bundle exec rake 'crawlscope:validate[https://example.com]'
+bundle exec rake crawlscope:validate:metadata URL=https://example.com
+bundle exec rake 'crawlscope:validate:metadata[https://example.com]'
+bundle exec rake crawlscope:validate:ldjson URL=https://example.com/article
+bundle exec rake 'crawlscope:validate:ldjson[https://example.com/article]'
+```
+
+`crawlscope:validate` runs all default sitemap rules: indexability, metadata,
+structured data, uniqueness, content quality, and links. `URL` is the site
+base. Without `SITEMAP`, Crawlscope uses the configured sitemap URL, then fetches
+`/sitemap.xml` from `URL` over HTTP. With `SITEMAP`, Crawlscope uses `URL` as
+the site base and validates URLs from that sitemap. `SITEMAP` may be a full URL
+or an explicitly selected local file path.
+
+Plain `rake` does not pass `--url` style flags to tasks. Use `URL=...` or the
+task-argument form above instead.
+
+`FETCH_EXECUTOR=async` is the default for HTTP crawling. It uses Ruby's fiber
+scheduler and Async::HTTP through Faraday, preserving the same `CONCURRENCY`
+bound. Use `FETCH_EXECUTOR=threaded` or `--fetch-executor threaded` for the
+thread-pool executor. Browser rendering uses the threaded executor by default
+because async fetch execution is only supported with HTTP rendering.
+
+`crawlscope:validate:ldjson` is separate because it directly checks the URL or semicolon-separated URLs in `URL`; it does not crawl the sitemap. Without `URL`, it checks the configured base URL, falling back to `http://localhost:3000`.
+
+## Structured Data URL Audit
+
+For one-off structured-data checks:
+
+```bash
+bin/rails crawlscope:validate:ldjson URL=https://example.com/article
+bin/rails crawlscope:validate:ldjson URL='https://example.com/a;https://example.com/b' SUMMARY=1
+bin/rails crawlscope:validate:ldjson URL=https://example.com/article REPORT_PATH=tmp/structured-data.json
+```
+
+Optional flags:
+
+- `DEBUG=1`: print detected items
+- `SUMMARY=1`: print grouped failures
+- `REPORT_PATH=...`: write a JSON report. Treat this as trusted operator input; Crawlscope writes to the path the task process can access.
+- `JS=1` or `RENDERER=browser`: render with Ferrum

data/docs/rules.md

@@ -0,0 +1,95 @@
+# Rules and schemas
+
+[README](../README.md) · [Project guide](https://www.ethos-link.com/opensource/crawlscope)
+
+Built-in rules:
+
+- `indexability`
+- `metadata`
+- `structured_data`
+- `uniqueness`
+- `content_quality`
+- `links`
+
+## Indexability
+
+Checks:
+
+- page-level meta robots `noindex`
+- `X-Robots-Tag: noindex`
+
+## Metadata
+
+Checks:
+
+- missing `<h1>`
+- missing `<title>`
+- title length
+- repeated site name in the title
+- missing meta description
+- meta description length
+- missing canonical link
+- canonical mismatch
+
+## Structured Data
+
+Checks:
+
+- malformed JSON-LD
+- missing required fields for supported schema types
+- schema validation failures from the configured registry
+- direct URL structured-data audits through `crawlscope:validate:ldjson`
+
+## Uniqueness
+
+Checks:
+
+- duplicate titles
+- duplicate meta descriptions
+- duplicate content fingerprints
+- near-duplicate visible content for up to 250 HTML pages
+
+For larger crawls, exact duplicate checks still run and Crawlscope reports
+`near_duplicate_scan_skipped`. Configure `Rules::Uniqueness` with
+`max_near_duplicate_pages:` in a custom rule registry to change the limit.
+
+## Content Quality
+
+Checks:
+
+- thin visible text
+- low visible-text-to-HTML ratio
+- low unique-token ratio
+
+## Links
+
+Checks:
+
+- broken internal links
+- unresolved internal links
+- low inbound anchor-link counts
+
+## Schema Registry
+
+`crawlscope` ships with a default schema registry for common types such as:
+
+- `Article`
+- `FAQPage`
+- `Organization`
+- `Product`
+- `Review`
+- `SoftwareApplication`
+- `WebApplication`
+- `WebSite`
+
+The default schema definitions live in `Crawlscope::Schemas`; `Crawlscope::SchemaRegistry` owns registration and validation.
+
+Host apps can replace or extend the registry:
+
+```ruby
+Crawlscope.configure do |config|
+  config.schema_registry = -> { MyApp::StructuredData::SchemaRegistry.new }
+end
+```
+
+That makes `crawlscope` useful as the audit engine while the app remains the owner of stricter product-specific schema rules.

data/docs/server-timing.md

@@ -0,0 +1,59 @@
+# Server timing
+
+[README](../README.md) · [Project guide](https://www.ethos-link.com/opensource/crawlscope)
+
+Crawlscope parses the
+[`Server-Timing`](https://www.w3.org/TR/server-timing/) response header for both
+HTTP and browser-rendered crawls. Each page exposes its parsed header through
+`page.server_timing`:
+
+```ruby
+page.server_timing.each do |metric|
+  puts [metric.name, metric.duration, metric.description].compact.join(": ")
+end
+```
+
+The validation report adds a `Server Timing` section only when at least one page
+publishes the header. It includes:
+
+- header coverage and the number of pages publishing durations
+- sample and page counts with average, p50, p95, and maximum per metric
+- non-duration signals, including cache status and routing descriptions
+- the ten pages with the largest individual metric
+- the number of malformed entries ignored during parsing
+
+`dur` values are reported as milliseconds, as recommended by the specification.
+Crawlscope does not add durations together because metrics such as `total`,
+`app`, and `db` may overlap. A page's worst-offender entry is its largest
+individual metric instead.
+
+The current HTTP and browser transports expose response headers, not response
+trailers. Metrics published only in a `Server-Timing` trailer are therefore not
+available to Crawlscope.
+
+Rails 8 applications can enable `ActionDispatch::ServerTiming` with:
+
+```ruby
+config.server_timing = true
+```
+
+Applications that expose timing only to authenticated diagnostics can set
+`config.profile_token`. When this value is present, Crawlscope sends it as
+`X-Profile-Token` on same-origin sitemap, HTTP, and browser document requests.
+The header survives same-origin redirects but is removed before a cross-origin
+redirect, and browser subresources never receive it. Keep the token in the host
+application's secret store rather than a URL, command argument, report, or
+checked-in configuration.
+
+`CRAWLSCOPE_PROFILE_TOKEN` is the portable default for Rails, the standalone
+CLI, Rake tasks, and plain Ruby callers. Rails applications may instead assign
+`config.profile_token` from encrypted credentials when that is their established
+secret-management boundary; explicit configuration takes precedence over the
+environment.
+
+New Rails applications enable Server-Timing in development by default;
+production remains opt-in. Rails publishes the Active Support notification names observed during
+each request and sums repeated events with the same name. The exact metrics
+therefore depend on the application and request, but commonly include controller,
+view, database, and cache instrumentation. Crawlscope accepts Rails' dotted
+metric names and reports each metric independently.

data/lib/crawlscope/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Crawlscope
-  VERSION = "0.10.0"
+  VERSION = "0.10.1"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: crawlscope
 version: !ruby/object:Gem::Version
-  version: 0.10.0
+  version: 0.10.1
 platform: ruby
 authors:
 - Paulo Fidalgo
@@ -178,6 +178,9 @@ files:
 - LICENSE.txt
 - README.md
 - UPGRADE.md
+- docs/rails.md
+- docs/rules.md
+- docs/server-timing.md
 - exe/crawlscope
 - lib/crawlscope.rb
 - lib/crawlscope/browser.rb
@@ -270,7 +273,7 @@ metadata:
   source_code_uri: https://github.com/ethos-link/crawlscope
   bug_tracker_uri: https://github.com/ethos-link/crawlscope/issues
   changelog_uri: https://github.com/ethos-link/crawlscope/blob/main/CHANGELOG.md
-  documentation_uri: https://github.com/ethos-link/crawlscope/blob/main/README.md
+  documentation_uri: https://www.ethos-link.com/opensource/crawlscope
   funding_uri: https://www.reviato.com/
   github_repo: ssh://github.com/ethos-link/crawlscope
   allowed_push_host: https://rubygems.org