crawlscope 0.8.1 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 89be661c49b5f2148a9d7962d81d522eedaedbba89c1f9dcf8474d62cfffd5f2
-  data.tar.gz: d27a26792cc5cfdab191371c3d005cf43298b02896cb7cbdf2113a5f341e6c12
+  metadata.gz: 94c540760a23969d9abd10c27866c6c14a7caf520a261e2f6a213ffabe5440de
+  data.tar.gz: f1caacfe42dc2e4cb398d1732b1a265761c0aabffc951647a93972479501835d
 SHA512:
-  metadata.gz: 5c90219d9430a09a3c95c5cb6367817cbc996f4ba69857781ddf6d2050373c9bfef2ad3de545a328542fa7362494df05be85cfa2c5e11f359b84870c8bee2ea6
-  data.tar.gz: 632ddff7f77d1c0eb62e6f9db0c9a36f8c3c080d2ac3b6b93bc03fcea65453d916ecbf7b39eb34e9b16d750309e7814586e2a84b4cd0157609a47402bcccf8ef
+  metadata.gz: e24e81f82b40938cb48182aef3657b3d49a273269900a5d47edeccd35f34081293044b1f9e0fb7e5bd6fbad26d18b0ec0f897b21b2935031d0601b081f1c0f2b
+  data.tar.gz: af73839632a975bc34145dbb821a02d953175cd5d6fa7c0af419ba7f488e02d043a946aeea8c338de1e338bc6864744a3a6ac23bbcaed6fa3ecf708ff2a4005c

data/CHANGELOG.md

@@ -5,6 +5,23 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.9.0] - 2026-11-04
+
+### Changed
+
+- Require Ruby 4.0 or newer.
+- Use published Gemkit 0.1.0 for shared verification and release PRs, with workflow callers pinned to the same release commit.
+
+### Added
+
+- Support optional host-installed Rails 8 (`>= 8.0`, `< 9`), including explicit `crawlscope/rails` loading after the core.
+- Reuse all packaged audit tasks in the repository, including indexability and content-quality checks.
+
+### Removed
+
+- Remove Railties as a runtime dependency so the Ruby library and CLI work without Rails.
+- Replace local tagging and hook bypasses with reviewed release PRs published by GitHub Actions.
+
 ## [0.8.1] - 2026-10-06
 
 

data/README.md

@@ -32,7 +32,8 @@ The default rule set includes:
 
 ## Installation
 
-Crawlscope requires Ruby 3.3 or newer.
+Crawlscope requires Ruby 4.0 or newer. Rails is optional; the CLI and Ruby
+library run without it. Rails integration requires host-installed Rails 8.
 
 Add this line to your application's Gemfile:
 
@@ -222,6 +223,10 @@ metric names and reports each metric independently.
 
 ## Rails Usage
 
+Install Rails 8 in the host application; Crawlscope does not install Railties.
+For a process that loads Crawlscope before Rails, require `crawlscope/rails`
+after Rails is available. The Ruby and CLI entry points remain independent of Rails.
+
 Run the install generator after adding the gem:
 
 ```bash
@@ -432,68 +437,92 @@ That makes `crawlscope` useful as the audit engine while the app remains the own
 
 ## Development
 
-```bash
+Shared development tooling comes from published `ethos-gemkit` 0.1.0. The Gemfile
+and workflow callers use the same release: the gem version is exact, and Actions
+references pin its full commit SHA.
+
+```sh
 git clone https://github.com/ethos-link/crawlscope.git
 cd crawlscope
-
 bundle install
-bundle exec rake test
-bundle exec rake standard
-bundle exec rake
+bundle exec lefthook install
+bundle exec rake verify
 ```
 
-### Git hooks
+`rake verify` runs Standard, all Minitest tests, package validation and a git-cliff
+smoke check. `rake` runs the same checks. Individual `rake test` and `rake standard`
+tasks remain available, along with the `crawlscope:*` audit tasks above.
 
-We use [lefthook](https://lefthook.dev/) with the Ruby
-[commitlint](https://github.com/arandilopez/commitlint) gem to enforce
-Conventional Commits on every commit. We also use
-[Standard Ruby](https://standardrb.com/) to keep code style consistent. CI
-validates commit messages, Standard Ruby, tests, and git-cliff changelog
-generation on pull requests and pushes to main/master.
+Hooks use Gemkit to block direct commits to main/master and validate Conventional
+Commits. CI uses the same lint, test, package, changelog and commit checks.
+GitLab sync has no caller because no target URL is configured.
 
-Run the hook installer once per clone:
+To install a development build locally, run `bundle exec rake install`.
 
-```bash
-bundle exec lefthook install
-```
+## Release
 
-### Install locally
+Use `release:prepare` from clean, up-to-date `main`. It creates a conventional
+release branch and review PR. **Merging the release PR starts publication through
+GitHub Actions**; ordinary development PRs run verification only.
 
-```bash
-rake install
-```
+### RubyGems setup
 
-## Release
+Configure **crawlscope → Trusted publishers** on RubyGems with these values:
 
-Releases are tag-driven and published by GitHub Actions to RubyGems.
-Local release commands never publish directly.
+| RubyGems field | Value |
+|---|---|
+| Repository owner | `ethos-link` |
+| Repository name | `crawlscope` |
+| Workflow filename | `release.yml` |
+| Environment | `rubygems` |
+| Workflow Repository Owner | `ethos-link` |
+| Workflow Repository Name | `ethos-gemkit` |
 
-Install [git-cliff](https://git-cliff.org/) locally before preparing a
-release. The release task prepends the next `CHANGELOG.md` section from
-Conventional Commits.
+The Workflow Repository fields are required because the publishing workflow lives
+in Gemkit. Gemkit's own publisher does not authorize Crawlscope. The account ID
+passed by the caller does not create publisher authorization.
 
-Before preparing a release, make sure you are on `main` or `master` with a
-clean worktree. If the release contains a breaking public-contract change,
-update `UPGRADE.md` with the host-app migration steps first.
+Install Git, git-cliff and GitHub CLI, authenticate with `gh auth login`, and
+install hooks as shown above. The `rubygems` GitHub environment must allow main
+and version tags; repository rules must permit the workflow's annotated tags.
 
-Then run one of:
+### Prepare and publish
 
-```bash
+Update `UPGRADE.md` with host-app migration steps for contract changes before
+preparing a release. Then run:
+
+```sh
+git switch main
+git pull --ff-only
+bundle install
 bundle exec rake 'release:prepare[patch]'
-bundle exec rake 'release:prepare[minor]'
-bundle exec rake 'release:prepare[major]'
-bundle exec rake 'release:prepare[0.1.0]'
 ```
 
-The task will:
+The default target is `patch`; `minor`, `major` and explicit versions are also
+supported. The task updates the version, changelog and lockfile, verifies the gem,
+commits with hooks enabled, pushes `chore/release-vX.Y.Z` and opens a release PR.
+Review its metadata and notes, and wait for CI before merging.
 
-1. Prepend the next `CHANGELOG.md` section with `git-cliff`.
-1. Update `lib/crawlscope/version.rb`.
-1. Commit the release changes.
-1. Create and push the `vX.Y.Z` tag.
+The workflow verifies and builds the exact release merge commit, creates an
+annotated tag, publishes the same artifact and creates GitHub release notes.
+No local publication or manual tag step is needed. Existing manually pushed
+annotated version tags remain supported.
+
+### Failed publication
+
+If credential setup reports `No trusted publisher configured for this workflow`
+and `gem push` was skipped, correct the publisher fields above and rerun only
+failed jobs in the original run:
+
+```sh
+gh run rerun RUN_ID --repo ethos-link/crawlscope --failed
+```
 
-The `Release` workflow then runs tests, publishes the gem to RubyGems,
-and creates the GitHub release from the changelog entry.
+The existing verified artifact and tag are reused. Do not run `release:prepare`
+again, bump the version or replace the tag to recover from that authentication
+failure. Failed release notes also use a failed-jobs retry without republishing.
+For uploads with uncertain results, already-published versions or expired
+artifacts, follow the [Gemkit recovery guide](https://github.com/ethos-link/ethos-gemkit#failures-and-retries).
 
 ## Contributing
 

data/UPGRADE.md

@@ -0,0 +1,138 @@
+# Upgrade Guide
+
+Use this file for host-app migration notes when a release changes public
+contracts, required setup, component locals, generated assets, or runtime
+behavior.
+
+## 0.9.0
+
+### Ruby 4 and optional Rails 8
+
+Upgrade Ruby to 4.0 or newer before updating Crawlscope. Rails integrations now
+require Rails 8 (`>= 8.0`, `< 9`). Upgrade the host Rails application when using
+the generator or Railtie; plain Ruby and CLI callers do not need Rails.
+
+Railties is no longer a runtime dependency. Rails applications must declare
+Rails or Railties themselves. If Crawlscope loads before Rails, explicitly
+require `crawlscope/rails` after the framework is available. Existing generated
+initializers and `require "crawlscope/tasks"` loaders remain supported.
+
+### Release preparation
+
+Development uses published Gemkit 0.1.0. Install the bundle and hooks with
+`bundle install` and `bundle exec lefthook install`.
+Run `release:prepare` from current main to open a verified release PR; merging
+that PR triggers workflow publication. Local preparation no longer tags, pushes
+main or bypasses hooks. Configure Crawlscope's RubyGems publisher as documented
+in the README; the shared workflow does not inherit another gem's permission.
+
+## Next Release
+
+### Structured-data length errors include the measured size
+
+String-length schema errors now include an optional `length` field with the
+actual character count. Text reports show the measurement and limit, such as
+`Article.headline: 125 characters; maximum 110`. Existing `field`, `issue`, and
+`type` fields remain available, and `issue` retains the original validator
+message. Applications that enforce exact error-object keys must accept `length`.
+
+### One warning per redirect or canonical mismatch
+
+Sitemap redirects now emit only `sitemap_redirect_url`, rather than also emitting
+`redirected_page`. Applications that match `redirected_page` must switch to
+`sitemap_redirect_url`; `final_url` and final HTTP `status` remain in details.
+
+Metadata and canonical-target checks skip redirect aliases, so the destination's
+canonical is not compared with the redirect's original URL. Metadata checks still
+run when the destination is itself in the crawl.
+
+A directly served page with a different canonical emits one warning:
+`non_canonical_page_in_sitemap` when it is in the sitemap, or `canonical_mismatch`
+otherwise. Applications that previously counted both warnings should handle
+either code as one mismatch.
+
+Metadata length reports put the field and limit in the group heading; rows show
+only the measured length. Server timing's page list now includes only durations
+strictly greater than 50 ms. Aggregate timing statistics still include all
+responses, and `result.server_timing_summary` retains all timing data.
+
+### Redirect URLs are excluded from uniqueness checks
+
+Uniqueness checks now compare only HTML pages whose normalized requested and
+final URLs match. Redirect aliases no longer produce duplicate-title,
+duplicate-description, duplicate-text, or similar-text warnings from the
+destination's content. Redirect issues remain in the crawl report. Separate
+pages that serve identical content without a redirect are still checked.
+
+No host-app configuration changes are required. Applications that count
+uniqueness issues should expect fewer warnings when sitemap URLs redirect.
+
+### Clearer audit messages
+
+Text reports now use plain-language group headings with severity counts and
+show errors before warnings and notices. The repeated category summary is
+removed. Structured-data errors show the type, field, and constraint on separate
+lines (for example, `Article.headline: 125 characters; maximum 110`). Sitemap
+redirects show the destination and its final HTTP status once, without the host
+for same-origin URLs.
+Canonical warnings now identify the canonical target and explain the mismatch.
+Link warnings use `linked from` and `links to` to identify each URL's role.
+Length and ratio warnings include the limit and units. Reports show every issue
+and every related URL on separate lines. Duplicate-title and description values
+are shown in full. Link rules retain all source and target URLs rather than
+three samples. Missing-sitemap issues keep `source_url` and also provide
+`source_urls` with all pages that link to the missing URL. Server timing retains
+full descriptions and lists every page whose duration exceeds 50 ms.
+
+Host applications that parse report text or issue messages must update their
+matching patterns. Prefer issue codes and structured `details` for automation;
+these retain their existing fields, including the original schema errors,
+redirect status and final URL, and canonical href. No configuration changes are
+required.
+
+The former report limits `Reporter::MAX_ISSUES_PER_GROUP` and
+`Rules::Links::MAX_SOURCES_IN_ERROR` are removed. Host applications that reference
+these constants must remove those references.
+
+### Default sitemap is fetched over HTTP
+
+Crawlscope no longer prefers `public/sitemap.xml` when validating a localhost
+application. Without an explicit `SITEMAP` or configured `sitemap_path`, it now
+fetches `/sitemap.xml` from the configured base URL so the crawl observes the
+live application and database state.
+
+The Rails installer now generates the same HTTP default. Regenerate or update
+existing initializers that use `Rails.public_path.join("sitemap.xml")`.
+Explicit local sitemap paths remain supported through `SITEMAP`, `--sitemap`,
+or `config.sitemap_path`.
+
+`CRAWLSCOPE_PROFILE_TOKEN` is now the portable default for standalone CLI, Rake,
+Rails, and plain Ruby usage. Explicit `config.profile_token` values still take
+precedence.
+
+### Server-Timing report output
+
+No host configuration is required. When one or more responses publish a
+`Server-Timing` header, the text report now includes an optional `Server Timing`
+section. Host applications that parse report text should accept this additional
+section.
+
+Parsed metrics are available through `page.server_timing`, and aggregate data is
+available through `result.server_timing_summary`. Crawlscope interprets `dur`
+values as milliseconds and ignores malformed entries while reporting their
+count.
+
+### Ruby 3.3 is now required
+
+Crawlscope now depends on the current Async runtime for production async HTTP
+fetching. Host applications must run Ruby 3.3 or newer before upgrading.
+
+Recommended migration:
+
+1. Upgrade the host application runtime to Ruby 3.3 or newer.
+2. Run `bundle update crawlscope async async-http async-http-faraday`.
+3. Crawlscope now uses `FETCH_EXECUTOR=async` by default for HTTP crawling.
+4. Set `FETCH_EXECUTOR=threaded` or pass `--fetch-executor threaded` for a
+   conservative rollout or for explicit thread-pool execution.
+5. Browser rendering continues to use threaded execution by default because
+   async fetch execution is only supported with HTTP rendering.

data/lib/crawlscope/rails.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+require "rails"
+require "rails/railtie"
+require "crawlscope"
+require "crawlscope/railtie"

data/lib/crawlscope/railtie.rb

@@ -1,5 +1,12 @@
 # frozen_string_literal: true
 
+require "rails"
+require "rails/railtie"
+
+unless Gem::Requirement.new(">= 8.0", "< 9").satisfied_by?(Gem.loaded_specs.fetch("railties").version)
+  raise LoadError, "crawlscope integration requires Rails 8.0 or newer within Rails 8"
+end
+
 module Crawlscope
   class Railtie < Rails::Railtie
     rake_tasks do

data/lib/crawlscope/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Crawlscope
-  VERSION = "0.8.1"
+  VERSION = "0.9.0"
 end

data/lib/crawlscope.rb

@@ -29,7 +29,7 @@ end
 Crawlscope.instance_variable_set(:@loader, Zeitwerk::Loader.for_gem)
 Crawlscope.loader.ignore("#{__dir__}/generators")
 Crawlscope.loader.ignore("#{__dir__}/tasks")
-Crawlscope.loader.ignore("#{__dir__}/crawlscope/railtie.rb")
+Crawlscope.loader.ignore("#{__dir__}/crawlscope/railtie.rb", "#{__dir__}/crawlscope/rails.rb")
 Crawlscope.loader.ignore("#{__dir__}/crawlscope/tasks.rb")
 Crawlscope.loader.setup
 

data/test/crawlscope/loader_test.rb

@@ -15,6 +15,7 @@ class CrawlscopeLoaderTest < Minitest::Test
     script = <<~RUBY
       require "crawlscope"
 
+      abort "Rails loaded by the core" if defined?(Rails)
       loaded = $LOADED_FEATURES.grep(%r{/(?:async|faraday|json-schema|nokogiri)(?:/|\\.rb)})
       abort loaded.join("\\n") unless loaded.empty?
     RUBY
@@ -29,6 +30,30 @@ class CrawlscopeLoaderTest < Minitest::Test
     assert status.success?, stderr
   end
 
+  def test_explicit_rails_integration_loads_after_the_core_and_registers_tasks
+    script = <<~RUBY
+      require "crawlscope"
+      require "crawlscope/rails"
+
+      abort "Railtie missing" unless Crawlscope::Railtie < Rails::Railtie
+      app = Class.new(Rails::Application)
+      app.config.eager_load = false
+      app.load_tasks
+      %w[validate validate:ldjson validate:indexability validate:content_quality].each do |task|
+        abort "task missing: \#{task}" unless Rake::Task.task_defined?("crawlscope:\#{task}")
+      end
+    RUBY
+
+    _stdout, stderr, status = Open3.capture3(
+      RbConfig.ruby,
+      "-I#{File.expand_path("../../lib", __dir__)}",
+      "-e",
+      script
+    )
+
+    assert status.success?, stderr
+  end
+
   def test_task_entrypoint_registers_tasks
     script = <<~RUBY
       require "crawlscope/tasks"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: crawlscope
 version: !ruby/object:Gem::Version
-  version: 0.8.1
+  version: 0.9.0
 platform: ruby
 authors:
 - Paulo Fidalgo
@@ -108,20 +108,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '1.16'
-- !ruby/object:Gem::Dependency
-  name: railties
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '7.1'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '7.1'
 - !ruby/object:Gem::Dependency
   name: zeitwerk
   requirement: !ruby/object:Gem::Requirement
@@ -150,20 +136,6 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '5.0'
-- !ruby/object:Gem::Dependency
-  name: rake
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '13.0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '13.0'
 - !ruby/object:Gem::Dependency
   name: simplecov
   requirement: !ruby/object:Gem::Requirement
@@ -178,20 +150,6 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '0.22'
-- !ruby/object:Gem::Dependency
-  name: standard
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.0'
 - !ruby/object:Gem::Dependency
   name: webmock
   requirement: !ruby/object:Gem::Requirement
@@ -219,6 +177,7 @@ files:
 - CHANGELOG.md
 - LICENSE.txt
 - README.md
+- UPGRADE.md
 - exe/crawlscope
 - lib/crawlscope.rb
 - lib/crawlscope/browser.rb
@@ -235,6 +194,7 @@ files:
 - lib/crawlscope/issue.rb
 - lib/crawlscope/issue_collection.rb
 - lib/crawlscope/page.rb
+- lib/crawlscope/rails.rb
 - lib/crawlscope/railtie.rb
 - lib/crawlscope/rake_tasks.rb
 - lib/crawlscope/reporter.rb
@@ -301,7 +261,6 @@ files:
 - test/performance/async_fetch_benchmark.rb
 - test/performance/fetch_executor_matrix.rb
 - test/performance/sitemap_expansion_benchmark.rb
-- test/release_task_test.rb
 - test/test_helper.rb
 homepage: https://www.ethos-link.com/opensource/crawlscope
 licenses:
@@ -323,7 +282,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: 3.3.0
+      version: '4.0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="

data/test/release_task_test.rb

@@ -1,86 +0,0 @@
-# frozen_string_literal: true
-
-require "test_helper"
-require "rake"
-
-unless respond_to?(:release_version, true)
-  load File.expand_path("../Rakefile", __dir__)
-end
-
-class ReleaseTaskTest < Minitest::Test
-  def test_release_version_increments_patch_from_current_version
-    major, minor, patch = Crawlscope::VERSION.split(".").map(&:to_i)
-
-    assert_equal "#{major}.#{minor}.#{patch + 1}", release_version("patch")
-  end
-
-  def test_release_version_accepts_explicit_semantic_version
-    assert_equal "0.3.0", release_version("0.3.0")
-  end
-
-  def test_validate_release_version_rejects_current_version
-    error = assert_raises(ArgumentError) do
-      validate_release_version!("0.2.7", "0.2.7")
-    end
-
-    assert_equal(
-      "Release version 0.2.7 must be newer than current version 0.2.7.",
-      error.message
-    )
-  end
-
-  def test_validate_release_version_rejects_existing_local_tag
-    @local_release_tag_exists = true
-    @remote_release_tag_exists = false
-
-    error = assert_raises(ArgumentError) do
-      validate_release_version!("0.2.8", "0.2.7")
-    end
-
-    assert_equal "Release tag v0.2.8 already exists locally.", error.message
-  end
-
-  def test_validate_release_version_rejects_existing_remote_tag
-    @local_release_tag_exists = false
-    @remote_release_tag_exists = true
-
-    error = assert_raises(ArgumentError) do
-      validate_release_version!("0.2.8", "0.2.7")
-    end
-
-    assert_equal "Release tag v0.2.8 already exists on origin.", error.message
-  end
-
-  def test_remote_release_tag_command_asks_git_to_fail_when_no_tag_matches
-    assert_equal(
-      "git ls-remote --exit-code --tags origin refs/tags/v0.2.8",
-      remote_release_tag_command("v0.2.8")
-    )
-  end
-
-  def test_changelog_command_prepends_the_next_release
-    assert_equal(
-      [
-        "git-cliff",
-        "-c",
-        "cliff.toml",
-        "--unreleased",
-        "--tag",
-        "v0.2.8",
-        "--prepend",
-        "CHANGELOG.md"
-      ],
-      changelog_command("0.2.8")
-    )
-  end
-
-  private
-
-  def local_release_tag_exists?(_tag)
-    @local_release_tag_exists || false
-  end
-
-  def remote_release_tag_exists?(_tag)
-    @remote_release_tag_exists || false
-  end
-end