crawlscope 0.8.1 → 0.9.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 89be661c49b5f2148a9d7962d81d522eedaedbba89c1f9dcf8474d62cfffd5f2
-  data.tar.gz: d27a26792cc5cfdab191371c3d005cf43298b02896cb7cbdf2113a5f341e6c12
+  metadata.gz: 98bf07eeaad6b7d357c0df661e1c2194b8c8dbfff9a78aeba1df98e99b3bba0d
+  data.tar.gz: f6a1c2848ffce1bae4175be567788f853df6f2fc583e324b7fbe6258429a8471
 SHA512:
-  metadata.gz: 5c90219d9430a09a3c95c5cb6367817cbc996f4ba69857781ddf6d2050373c9bfef2ad3de545a328542fa7362494df05be85cfa2c5e11f359b84870c8bee2ea6
-  data.tar.gz: 632ddff7f77d1c0eb62e6f9db0c9a36f8c3c080d2ac3b6b93bc03fcea65453d916ecbf7b39eb34e9b16d750309e7814586e2a84b4cd0157609a47402bcccf8ef
+  metadata.gz: be3fb58b969d7b46081c31b000048c6f58e934ad8d44444dd4d71d017cd5fd5119e6efd669bf024e7a769900942381bd6938edd434ed3d82754773b3f613578c
+  data.tar.gz: 9c9a67eee2092834bbb4bd6ac285e277dd6ef22dfcb1083fc9fc2aa2e39ace412376c2b0929306ff367cf1ba8f70e3cd787f31bbc2336217ac64d0e4de488e79

data/CHANGELOG.md

@@ -5,6 +5,32 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.9.1] - 2026-11-04
+
+
+### Documentation
+
+- adopt Gemkit 0.2 and shorten shared guidance
+
+
+
+## [0.9.0] - 2026-11-04
+
+### Changed
+
+- Require Ruby 4.0 or newer.
+- Use published Gemkit 0.1.0 for shared verification and release PRs, with workflow callers pinned to the same release commit.
+
+### Added
+
+- Support optional host-installed Rails 8 (`>= 8.0`, `< 9`), including explicit `crawlscope/rails` loading after the core.
+- Reuse all packaged audit tasks in the repository, including indexability and content-quality checks.
+
+### Removed
+
+- Remove Railties as a runtime dependency so the Ruby library and CLI work without Rails.
+- Replace local tagging and hook bypasses with reviewed release PRs published by GitHub Actions.
+
 ## [0.8.1] - 2026-10-06
 
 

data/README.md

@@ -32,7 +32,8 @@ The default rule set includes:
 
 ## Installation
 
-Crawlscope requires Ruby 3.3 or newer.
+Crawlscope requires Ruby 4.0 or newer. Rails is optional; the CLI and Ruby
+library run without it. Rails integration requires host-installed Rails 8.
 
 Add this line to your application's Gemfile:
 
@@ -222,6 +223,10 @@ metric names and reports each metric independently.
 
 ## Rails Usage
 
+Install Rails 8 in the host application; Crawlscope does not install Railties.
+For a process that loads Crawlscope before Rails, require `crawlscope/rails`
+after Rails is available. The Ruby and CLI entry points remain independent of Rails.
+
 Run the install generator after adding the gem:
 
 ```bash
@@ -430,71 +435,6 @@ end
 
 That makes `crawlscope` useful as the audit engine while the app remains the owner of stricter product-specific schema rules.
 
-## Development
-
-```bash
-git clone https://github.com/ethos-link/crawlscope.git
-cd crawlscope
-
-bundle install
-bundle exec rake test
-bundle exec rake standard
-bundle exec rake
-```
-
-### Git hooks
-
-We use [lefthook](https://lefthook.dev/) with the Ruby
-[commitlint](https://github.com/arandilopez/commitlint) gem to enforce
-Conventional Commits on every commit. We also use
-[Standard Ruby](https://standardrb.com/) to keep code style consistent. CI
-validates commit messages, Standard Ruby, tests, and git-cliff changelog
-generation on pull requests and pushes to main/master.
-
-Run the hook installer once per clone:
-
-```bash
-bundle exec lefthook install
-```
-
-### Install locally
-
-```bash
-rake install
-```
-
-## Release
-
-Releases are tag-driven and published by GitHub Actions to RubyGems.
-Local release commands never publish directly.
-
-Install [git-cliff](https://git-cliff.org/) locally before preparing a
-release. The release task prepends the next `CHANGELOG.md` section from
-Conventional Commits.
-
-Before preparing a release, make sure you are on `main` or `master` with a
-clean worktree. If the release contains a breaking public-contract change,
-update `UPGRADE.md` with the host-app migration steps first.
-
-Then run one of:
-
-```bash
-bundle exec rake 'release:prepare[patch]'
-bundle exec rake 'release:prepare[minor]'
-bundle exec rake 'release:prepare[major]'
-bundle exec rake 'release:prepare[0.1.0]'
-```
-
-The task will:
-
-1. Prepend the next `CHANGELOG.md` section with `git-cliff`.
-1. Update `lib/crawlscope/version.rb`.
-1. Commit the release changes.
-1. Create and push the `vX.Y.Z` tag.
-
-The `Release` workflow then runs tests, publishes the gem to RubyGems,
-and creates the GitHub release from the changelog entry.
-
 ## Contributing
 
 1. Fork it
@@ -509,9 +449,31 @@ Please use [Conventional Commits](https://www.conventionalcommits.org/) for comm
 
 MIT License, see [LICENSE.txt](LICENSE.txt)
 
-## About
+## Development and releases
+
+Use published [Ethos Gemkit](https://github.com/ethos-link/ethos-gemkit) for shared
+tooling. Keep the exact Gemfile version and workflow commit pins on the same release.
+
+```sh
+bundle install
+bundle exec lefthook install
+bundle exec rake verify
+```
+
+Tests use controlled local fixtures and need no service account. From clean,
+current main, run `bundle exec rake 'release:prepare[patch]'`; review and merge
+the generated PR to publish through Actions. Shared prerequisites, publisher
+settings and failed-job recovery are in the [Gemkit guide](https://github.com/ethos-link/ethos-gemkit#releasing-a-gem).
+
+## Built by Ethos Link
 
-Made by the team at [Ethos Link](https://www.ethos-link.com) — practical software for growing businesses. We build tools for hospitality operators who need clear workflows, fast onboarding, and real human support.
+[Ethos Link](https://www.ethos-link.com) builds business software and integrations
+that replace manual handoffs with connected workflows. We help teams automate
+routine work, use their data and improve the processes their business runs on.
 
-We also build [Reviato](https://www.reviato.com), “Capture. Interpret. Act.”.
-Turn guest feedback into clear next steps for your team. Collect private appraisals, spot patterns across reviews, and act before small issues turn into public ones.
+- [Reviato](https://www.reviato.com): Read public review trends and collect
+  private customer feedback, so your team can decide what to improve.
+- [EthosPlaces](https://www.ethos.place): Give customers the details they need to
+  choose and visit your business, from opening hours to directions.
+- [Nuntio](https://www.nuntiohq.com): Draft, review and schedule social posts in
+  one workspace, then track each publishing result.

data/UPGRADE.md

@@ -0,0 +1,144 @@
+# Upgrade Guide
+
+## 0.9.1
+
+Development and release tooling uses Gemkit 0.2.0. Update the gem and workflow
+pins together; see the [Gemkit upgrade guide](https://github.com/ethos-link/ethos-gemkit/blob/main/UPGRADE.md).
+Runtime APIs are unchanged.
+
+Use this file for host-app migration notes when a release changes public
+contracts, required setup, component locals, generated assets, or runtime
+behavior.
+
+## 0.9.0
+
+### Ruby 4 and optional Rails 8
+
+Upgrade Ruby to 4.0 or newer before updating Crawlscope. Rails integrations now
+require Rails 8 (`>= 8.0`, `< 9`). Upgrade the host Rails application when using
+the generator or Railtie; plain Ruby and CLI callers do not need Rails.
+
+Railties is no longer a runtime dependency. Rails applications must declare
+Rails or Railties themselves. If Crawlscope loads before Rails, explicitly
+require `crawlscope/rails` after the framework is available. Existing generated
+initializers and `require "crawlscope/tasks"` loaders remain supported.
+
+### Release preparation
+
+Development uses published Gemkit 0.1.0. Install the bundle and hooks with
+`bundle install` and `bundle exec lefthook install`.
+Run `release:prepare` from current main to open a verified release PR; merging
+that PR triggers workflow publication. Local preparation no longer tags, pushes
+main or bypasses hooks. Configure Crawlscope's RubyGems publisher as documented
+in the README; the shared workflow does not inherit another gem's permission.
+
+## Next Release
+
+### Structured-data length errors include the measured size
+
+String-length schema errors now include an optional `length` field with the
+actual character count. Text reports show the measurement and limit, such as
+`Article.headline: 125 characters; maximum 110`. Existing `field`, `issue`, and
+`type` fields remain available, and `issue` retains the original validator
+message. Applications that enforce exact error-object keys must accept `length`.
+
+### One warning per redirect or canonical mismatch
+
+Sitemap redirects now emit only `sitemap_redirect_url`, rather than also emitting
+`redirected_page`. Applications that match `redirected_page` must switch to
+`sitemap_redirect_url`; `final_url` and final HTTP `status` remain in details.
+
+Metadata and canonical-target checks skip redirect aliases, so the destination's
+canonical is not compared with the redirect's original URL. Metadata checks still
+run when the destination is itself in the crawl.
+
+A directly served page with a different canonical emits one warning:
+`non_canonical_page_in_sitemap` when it is in the sitemap, or `canonical_mismatch`
+otherwise. Applications that previously counted both warnings should handle
+either code as one mismatch.
+
+Metadata length reports put the field and limit in the group heading; rows show
+only the measured length. Server timing's page list now includes only durations
+strictly greater than 50 ms. Aggregate timing statistics still include all
+responses, and `result.server_timing_summary` retains all timing data.
+
+### Redirect URLs are excluded from uniqueness checks
+
+Uniqueness checks now compare only HTML pages whose normalized requested and
+final URLs match. Redirect aliases no longer produce duplicate-title,
+duplicate-description, duplicate-text, or similar-text warnings from the
+destination's content. Redirect issues remain in the crawl report. Separate
+pages that serve identical content without a redirect are still checked.
+
+No host-app configuration changes are required. Applications that count
+uniqueness issues should expect fewer warnings when sitemap URLs redirect.
+
+### Clearer audit messages
+
+Text reports now use plain-language group headings with severity counts and
+show errors before warnings and notices. The repeated category summary is
+removed. Structured-data errors show the type, field, and constraint on separate
+lines (for example, `Article.headline: 125 characters; maximum 110`). Sitemap
+redirects show the destination and its final HTTP status once, without the host
+for same-origin URLs.
+Canonical warnings now identify the canonical target and explain the mismatch.
+Link warnings use `linked from` and `links to` to identify each URL's role.
+Length and ratio warnings include the limit and units. Reports show every issue
+and every related URL on separate lines. Duplicate-title and description values
+are shown in full. Link rules retain all source and target URLs rather than
+three samples. Missing-sitemap issues keep `source_url` and also provide
+`source_urls` with all pages that link to the missing URL. Server timing retains
+full descriptions and lists every page whose duration exceeds 50 ms.
+
+Host applications that parse report text or issue messages must update their
+matching patterns. Prefer issue codes and structured `details` for automation;
+these retain their existing fields, including the original schema errors,
+redirect status and final URL, and canonical href. No configuration changes are
+required.
+
+The former report limits `Reporter::MAX_ISSUES_PER_GROUP` and
+`Rules::Links::MAX_SOURCES_IN_ERROR` are removed. Host applications that reference
+these constants must remove those references.
+
+### Default sitemap is fetched over HTTP
+
+Crawlscope no longer prefers `public/sitemap.xml` when validating a localhost
+application. Without an explicit `SITEMAP` or configured `sitemap_path`, it now
+fetches `/sitemap.xml` from the configured base URL so the crawl observes the
+live application and database state.
+
+The Rails installer now generates the same HTTP default. Regenerate or update
+existing initializers that use `Rails.public_path.join("sitemap.xml")`.
+Explicit local sitemap paths remain supported through `SITEMAP`, `--sitemap`,
+or `config.sitemap_path`.
+
+`CRAWLSCOPE_PROFILE_TOKEN` is now the portable default for standalone CLI, Rake,
+Rails, and plain Ruby usage. Explicit `config.profile_token` values still take
+precedence.
+
+### Server-Timing report output
+
+No host configuration is required. When one or more responses publish a
+`Server-Timing` header, the text report now includes an optional `Server Timing`
+section. Host applications that parse report text should accept this additional
+section.
+
+Parsed metrics are available through `page.server_timing`, and aggregate data is
+available through `result.server_timing_summary`. Crawlscope interprets `dur`
+values as milliseconds and ignores malformed entries while reporting their
+count.
+
+### Ruby 3.3 is now required
+
+Crawlscope now depends on the current Async runtime for production async HTTP
+fetching. Host applications must run Ruby 3.3 or newer before upgrading.
+
+Recommended migration:
+
+1. Upgrade the host application runtime to Ruby 3.3 or newer.
+2. Run `bundle update crawlscope async async-http async-http-faraday`.
+3. Crawlscope now uses `FETCH_EXECUTOR=async` by default for HTTP crawling.
+4. Set `FETCH_EXECUTOR=threaded` or pass `--fetch-executor threaded` for a
+   conservative rollout or for explicit thread-pool execution.
+5. Browser rendering continues to use threaded execution by default because
+   async fetch execution is only supported with HTTP rendering.

data/lib/crawlscope/rails.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+require "rails"
+require "rails/railtie"
+require "crawlscope"
+require "crawlscope/railtie"

data/lib/crawlscope/railtie.rb

@@ -1,5 +1,12 @@
 # frozen_string_literal: true
 
+require "rails"
+require "rails/railtie"
+
+unless Gem::Requirement.new(">= 8.0", "< 9").satisfied_by?(Gem.loaded_specs.fetch("railties").version)
+  raise LoadError, "crawlscope integration requires Rails 8.0 or newer within Rails 8"
+end
+
 module Crawlscope
   class Railtie < Rails::Railtie
     rake_tasks do

data/lib/crawlscope/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Crawlscope
-  VERSION = "0.8.1"
+  VERSION = "0.9.1"
 end

data/lib/crawlscope.rb

@@ -29,7 +29,7 @@ end
 Crawlscope.instance_variable_set(:@loader, Zeitwerk::Loader.for_gem)
 Crawlscope.loader.ignore("#{__dir__}/generators")
 Crawlscope.loader.ignore("#{__dir__}/tasks")
-Crawlscope.loader.ignore("#{__dir__}/crawlscope/railtie.rb")
+Crawlscope.loader.ignore("#{__dir__}/crawlscope/railtie.rb", "#{__dir__}/crawlscope/rails.rb")
 Crawlscope.loader.ignore("#{__dir__}/crawlscope/tasks.rb")
 Crawlscope.loader.setup
 

data/test/crawlscope/loader_test.rb

@@ -15,6 +15,7 @@ class CrawlscopeLoaderTest < Minitest::Test
     script = <<~RUBY
       require "crawlscope"
 
+      abort "Rails loaded by the core" if defined?(Rails)
       loaded = $LOADED_FEATURES.grep(%r{/(?:async|faraday|json-schema|nokogiri)(?:/|\\.rb)})
       abort loaded.join("\\n") unless loaded.empty?
     RUBY
@@ -29,6 +30,30 @@ class CrawlscopeLoaderTest < Minitest::Test
     assert status.success?, stderr
   end
 
+  def test_explicit_rails_integration_loads_after_the_core_and_registers_tasks
+    script = <<~RUBY
+      require "crawlscope"
+      require "crawlscope/rails"
+
+      abort "Railtie missing" unless Crawlscope::Railtie < Rails::Railtie
+      app = Class.new(Rails::Application)
+      app.config.eager_load = false
+      app.load_tasks
+      %w[validate validate:ldjson validate:indexability validate:content_quality].each do |task|
+        abort "task missing: \#{task}" unless Rake::Task.task_defined?("crawlscope:\#{task}")
+      end
+    RUBY
+
+    _stdout, stderr, status = Open3.capture3(
+      RbConfig.ruby,
+      "-I#{File.expand_path("../../lib", __dir__)}",
+      "-e",
+      script
+    )
+
+    assert status.success?, stderr
+  end
+
   def test_task_entrypoint_registers_tasks
     script = <<~RUBY
       require "crawlscope/tasks"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: crawlscope
 version: !ruby/object:Gem::Version
-  version: 0.8.1
+  version: 0.9.1
 platform: ruby
 authors:
 - Paulo Fidalgo
@@ -108,20 +108,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '1.16'
-- !ruby/object:Gem::Dependency
-  name: railties
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '7.1'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '7.1'
 - !ruby/object:Gem::Dependency
   name: zeitwerk
   requirement: !ruby/object:Gem::Requirement
@@ -150,20 +136,6 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '5.0'
-- !ruby/object:Gem::Dependency
-  name: rake
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '13.0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '13.0'
 - !ruby/object:Gem::Dependency
   name: simplecov
   requirement: !ruby/object:Gem::Requirement
@@ -178,20 +150,6 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '0.22'
-- !ruby/object:Gem::Dependency
-  name: standard
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.0'
 - !ruby/object:Gem::Dependency
   name: webmock
   requirement: !ruby/object:Gem::Requirement
@@ -219,6 +177,7 @@ files:
 - CHANGELOG.md
 - LICENSE.txt
 - README.md
+- UPGRADE.md
 - exe/crawlscope
 - lib/crawlscope.rb
 - lib/crawlscope/browser.rb
@@ -235,6 +194,7 @@ files:
 - lib/crawlscope/issue.rb
 - lib/crawlscope/issue_collection.rb
 - lib/crawlscope/page.rb
+- lib/crawlscope/rails.rb
 - lib/crawlscope/railtie.rb
 - lib/crawlscope/rake_tasks.rb
 - lib/crawlscope/reporter.rb
@@ -301,7 +261,6 @@ files:
 - test/performance/async_fetch_benchmark.rb
 - test/performance/fetch_executor_matrix.rb
 - test/performance/sitemap_expansion_benchmark.rb
-- test/release_task_test.rb
 - test/test_helper.rb
 homepage: https://www.ethos-link.com/opensource/crawlscope
 licenses:
@@ -323,7 +282,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: 3.3.0
+      version: '4.0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="

data/test/release_task_test.rb

@@ -1,86 +0,0 @@
-# frozen_string_literal: true
-
-require "test_helper"
-require "rake"
-
-unless respond_to?(:release_version, true)
-  load File.expand_path("../Rakefile", __dir__)
-end
-
-class ReleaseTaskTest < Minitest::Test
-  def test_release_version_increments_patch_from_current_version
-    major, minor, patch = Crawlscope::VERSION.split(".").map(&:to_i)
-
-    assert_equal "#{major}.#{minor}.#{patch + 1}", release_version("patch")
-  end
-
-  def test_release_version_accepts_explicit_semantic_version
-    assert_equal "0.3.0", release_version("0.3.0")
-  end
-
-  def test_validate_release_version_rejects_current_version
-    error = assert_raises(ArgumentError) do
-      validate_release_version!("0.2.7", "0.2.7")
-    end
-
-    assert_equal(
-      "Release version 0.2.7 must be newer than current version 0.2.7.",
-      error.message
-    )
-  end
-
-  def test_validate_release_version_rejects_existing_local_tag
-    @local_release_tag_exists = true
-    @remote_release_tag_exists = false
-
-    error = assert_raises(ArgumentError) do
-      validate_release_version!("0.2.8", "0.2.7")
-    end
-
-    assert_equal "Release tag v0.2.8 already exists locally.", error.message
-  end
-
-  def test_validate_release_version_rejects_existing_remote_tag
-    @local_release_tag_exists = false
-    @remote_release_tag_exists = true
-
-    error = assert_raises(ArgumentError) do
-      validate_release_version!("0.2.8", "0.2.7")
-    end
-
-    assert_equal "Release tag v0.2.8 already exists on origin.", error.message
-  end
-
-  def test_remote_release_tag_command_asks_git_to_fail_when_no_tag_matches
-    assert_equal(
-      "git ls-remote --exit-code --tags origin refs/tags/v0.2.8",
-      remote_release_tag_command("v0.2.8")
-    )
-  end
-
-  def test_changelog_command_prepends_the_next_release
-    assert_equal(
-      [
-        "git-cliff",
-        "-c",
-        "cliff.toml",
-        "--unreleased",
-        "--tag",
-        "v0.2.8",
-        "--prepend",
-        "CHANGELOG.md"
-      ],
-      changelog_command("0.2.8")
-    )
-  end
-
-  private
-
-  def local_release_tag_exists?(_tag)
-    @local_release_tag_exists || false
-  end
-
-  def remote_release_tag_exists?(_tag)
-    @remote_release_tag_exists || false
-  end
-end