crawlscope 0.8.0 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4b830667c3f7647baba6ca372e78daa5f4116e2178fc585aff4f387671e71b81
-  data.tar.gz: 357a5ceab12f86addda14927ba03be7357f20d301ddeb58b72a87527133653ce
+  metadata.gz: 94c540760a23969d9abd10c27866c6c14a7caf520a261e2f6a213ffabe5440de
+  data.tar.gz: f1caacfe42dc2e4cb398d1732b1a265761c0aabffc951647a93972479501835d
 SHA512:
-  metadata.gz: 954a7f1044ae369e01fc571331acbf0b9ebd47f56d8047d028c6967a85cf5caefa673a3bfa05b782a814d3233f883ea13ca98a3ab961d2e8de47d788804f6158
-  data.tar.gz: 8a59e970c7191847a25b855e7342d0ebaab7b881540d41ac78b9b558460ffd9a679fbfff3c54ef78e22f2bd287d8009f9d58961128b9c4b74ac3773ce8dc82dd
+  metadata.gz: e24e81f82b40938cb48182aef3657b3d49a273269900a5d47edeccd35f34081293044b1f9e0fb7e5bd6fbad26d18b0ec0f897b21b2935031d0601b081f1c0f2b
+  data.tar.gz: af73839632a975bc34145dbb821a02d953175cd5d6fa7c0af419ba7f488e02d043a946aeea8c338de1e338bc6864744a3a6ac23bbcaed6fa3ecf708ff2a4005c

data/CHANGELOG.md

@@ -5,6 +5,32 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.9.0] - 2026-11-04
+
+### Changed
+
+- Require Ruby 4.0 or newer.
+- Use published Gemkit 0.1.0 for shared verification and release PRs, with workflow callers pinned to the same release commit.
+
+### Added
+
+- Support optional host-installed Rails 8 (`>= 8.0`, `< 9`), including explicit `crawlscope/rails` loading after the core.
+- Reuse all packaged audit tasks in the repository, including indexability and content-quality checks.
+
+### Removed
+
+- Remove Railties as a runtime dependency so the Ruby library and CLI work without Rails.
+- Replace local tagging and hook bypasses with reviewed release PRs published by GitHub Actions.
+
+## [0.8.1] - 2026-10-06
+
+
+### Fixed
+
+- clarify findings and prevent redirect false positives
+
+
+
 ## [0.8.0] - 2026-07-24
 
 

data/README.md

@@ -32,7 +32,8 @@ The default rule set includes:
 
 ## Installation
 
-Crawlscope requires Ruby 3.3 or newer.
+Crawlscope requires Ruby 4.0 or newer. Rails is optional; the CLI and Ruby
+library run without it. Rails integration requires host-installed Rails 8.
 
 Add this line to your application's Gemfile:
 
@@ -106,21 +107,32 @@ Base URL: https://example.com
 Sitemap: https://example.com/sitemap.xml
 URLs: 24
 Pages: 24
-Status: FAILED
-Issues: 3 3 warnings
+Status: WARNINGS
+Issues: 3 total (3 warnings)
 
-Summary:
-  links            2
-  metadata         1
+Links: Too few incoming links without nofollow (2 warnings)
+  - /pricing  incoming links without nofollow: 1; minimum 2
+      linked from: /
+  - /features  incoming links without nofollow: 1; minimum 2
+      linked from: /
 
-links / low_dofollow_inlinks: 2
-  - /pricing  inbound 1/2  sources: /
-  - /features  inbound 1/2  sources: /
-
-metadata / missing_title: 1
+Metadata: Missing title (1 warning)
   - /draft  missing <title>
 ```
 
+Reports include all issues and related URLs. Errors appear before warnings and
+notices. Each related URL has a label, such as `linked from`, `links to`, or
+`page`. JSON-LD errors appear below the page URL with the type, field, and
+constraint. Same-origin URLs use paths; other URLs retain their scheme and host.
+For automation in Ruby, use `result.issues` and each issue's `code`, `severity`,
+`url`, and `details` rather than matching the report text.
+
+Metadata length limits appear in the group heading; each row shows the measured
+length. The default maximums are 160 characters for a meta description and 110
+characters for an Article JSON-LD `headline`. These are separate Crawlscope
+validation limits for separate fields. Server timing lists pages only when the
+reported duration exceeds 50 ms; aggregate statistics include all responses.
+
 ## Ruby Usage
 
 ```ruby
@@ -211,6 +223,10 @@ metric names and reports each metric independently.
 
 ## Rails Usage
 
+Install Rails 8 in the host application; Crawlscope does not install Railties.
+For a process that loads Crawlscope before Rails, require `crawlscope/rails`
+after Rails is available. The Ruby and CLI entry points remain independent of Rails.
+
 Run the install generator after adding the gem:
 
 ```bash
@@ -421,68 +437,92 @@ That makes `crawlscope` useful as the audit engine while the app remains the own
 
 ## Development
 
-```bash
+Shared development tooling comes from published `ethos-gemkit` 0.1.0. The Gemfile
+and workflow callers use the same release: the gem version is exact, and Actions
+references pin its full commit SHA.
+
+```sh
 git clone https://github.com/ethos-link/crawlscope.git
 cd crawlscope
-
 bundle install
-bundle exec rake test
-bundle exec rake standard
-bundle exec rake
+bundle exec lefthook install
+bundle exec rake verify
 ```
 
-### Git hooks
+`rake verify` runs Standard, all Minitest tests, package validation and a git-cliff
+smoke check. `rake` runs the same checks. Individual `rake test` and `rake standard`
+tasks remain available, along with the `crawlscope:*` audit tasks above.
 
-We use [lefthook](https://lefthook.dev/) with the Ruby
-[commitlint](https://github.com/arandilopez/commitlint) gem to enforce
-Conventional Commits on every commit. We also use
-[Standard Ruby](https://standardrb.com/) to keep code style consistent. CI
-validates commit messages, Standard Ruby, tests, and git-cliff changelog
-generation on pull requests and pushes to main/master.
+Hooks use Gemkit to block direct commits to main/master and validate Conventional
+Commits. CI uses the same lint, test, package, changelog and commit checks.
+GitLab sync has no caller because no target URL is configured.
 
-Run the hook installer once per clone:
+To install a development build locally, run `bundle exec rake install`.
 
-```bash
-bundle exec lefthook install
-```
+## Release
 
-### Install locally
+Use `release:prepare` from clean, up-to-date `main`. It creates a conventional
+release branch and review PR. **Merging the release PR starts publication through
+GitHub Actions**; ordinary development PRs run verification only.
 
-```bash
-rake install
-```
+### RubyGems setup
 
-## Release
+Configure **crawlscope → Trusted publishers** on RubyGems with these values:
 
-Releases are tag-driven and published by GitHub Actions to RubyGems.
-Local release commands never publish directly.
+| RubyGems field | Value |
+|---|---|
+| Repository owner | `ethos-link` |
+| Repository name | `crawlscope` |
+| Workflow filename | `release.yml` |
+| Environment | `rubygems` |
+| Workflow Repository Owner | `ethos-link` |
+| Workflow Repository Name | `ethos-gemkit` |
 
-Install [git-cliff](https://git-cliff.org/) locally before preparing a
-release. The release task prepends the next `CHANGELOG.md` section from
-Conventional Commits.
+The Workflow Repository fields are required because the publishing workflow lives
+in Gemkit. Gemkit's own publisher does not authorize Crawlscope. The account ID
+passed by the caller does not create publisher authorization.
 
-Before preparing a release, make sure you are on `main` or `master` with a
-clean worktree. If the release contains a breaking public-contract change,
-update `UPGRADE.md` with the host-app migration steps first.
+Install Git, git-cliff and GitHub CLI, authenticate with `gh auth login`, and
+install hooks as shown above. The `rubygems` GitHub environment must allow main
+and version tags; repository rules must permit the workflow's annotated tags.
 
-Then run one of:
+### Prepare and publish
 
-```bash
+Update `UPGRADE.md` with host-app migration steps for contract changes before
+preparing a release. Then run:
+
+```sh
+git switch main
+git pull --ff-only
+bundle install
 bundle exec rake 'release:prepare[patch]'
-bundle exec rake 'release:prepare[minor]'
-bundle exec rake 'release:prepare[major]'
-bundle exec rake 'release:prepare[0.1.0]'
 ```
 
-The task will:
+The default target is `patch`; `minor`, `major` and explicit versions are also
+supported. The task updates the version, changelog and lockfile, verifies the gem,
+commits with hooks enabled, pushes `chore/release-vX.Y.Z` and opens a release PR.
+Review its metadata and notes, and wait for CI before merging.
+
+The workflow verifies and builds the exact release merge commit, creates an
+annotated tag, publishes the same artifact and creates GitHub release notes.
+No local publication or manual tag step is needed. Existing manually pushed
+annotated version tags remain supported.
+
+### Failed publication
 
-1. Prepend the next `CHANGELOG.md` section with `git-cliff`.
-1. Update `lib/crawlscope/version.rb`.
-1. Commit the release changes.
-1. Create and push the `vX.Y.Z` tag.
+If credential setup reports `No trusted publisher configured for this workflow`
+and `gem push` was skipped, correct the publisher fields above and rerun only
+failed jobs in the original run:
+
+```sh
+gh run rerun RUN_ID --repo ethos-link/crawlscope --failed
+```
 
-The `Release` workflow then runs tests, publishes the gem to RubyGems,
-and creates the GitHub release from the changelog entry.
+The existing verified artifact and tag are reused. Do not run `release:prepare`
+again, bump the version or replace the tag to recover from that authentication
+failure. Failed release notes also use a failed-jobs retry without republishing.
+For uploads with uncertain results, already-published versions or expired
+artifacts, follow the [Gemkit recovery guide](https://github.com/ethos-link/ethos-gemkit#failures-and-retries).
 
 ## Contributing
 

data/UPGRADE.md

@@ -0,0 +1,138 @@
+# Upgrade Guide
+
+Use this file for host-app migration notes when a release changes public
+contracts, required setup, component locals, generated assets, or runtime
+behavior.
+
+## 0.9.0
+
+### Ruby 4 and optional Rails 8
+
+Upgrade Ruby to 4.0 or newer before updating Crawlscope. Rails integrations now
+require Rails 8 (`>= 8.0`, `< 9`). Upgrade the host Rails application when using
+the generator or Railtie; plain Ruby and CLI callers do not need Rails.
+
+Railties is no longer a runtime dependency. Rails applications must declare
+Rails or Railties themselves. If Crawlscope loads before Rails, explicitly
+require `crawlscope/rails` after the framework is available. Existing generated
+initializers and `require "crawlscope/tasks"` loaders remain supported.
+
+### Release preparation
+
+Development uses published Gemkit 0.1.0. Install the bundle and hooks with
+`bundle install` and `bundle exec lefthook install`.
+Run `release:prepare` from current main to open a verified release PR; merging
+that PR triggers workflow publication. Local preparation no longer tags, pushes
+main or bypasses hooks. Configure Crawlscope's RubyGems publisher as documented
+in the README; the shared workflow does not inherit another gem's permission.
+
+## Next Release
+
+### Structured-data length errors include the measured size
+
+String-length schema errors now include an optional `length` field with the
+actual character count. Text reports show the measurement and limit, such as
+`Article.headline: 125 characters; maximum 110`. Existing `field`, `issue`, and
+`type` fields remain available, and `issue` retains the original validator
+message. Applications that enforce exact error-object keys must accept `length`.
+
+### One warning per redirect or canonical mismatch
+
+Sitemap redirects now emit only `sitemap_redirect_url`, rather than also emitting
+`redirected_page`. Applications that match `redirected_page` must switch to
+`sitemap_redirect_url`; `final_url` and final HTTP `status` remain in details.
+
+Metadata and canonical-target checks skip redirect aliases, so the destination's
+canonical is not compared with the redirect's original URL. Metadata checks still
+run when the destination is itself in the crawl.
+
+A directly served page with a different canonical emits one warning:
+`non_canonical_page_in_sitemap` when it is in the sitemap, or `canonical_mismatch`
+otherwise. Applications that previously counted both warnings should handle
+either code as one mismatch.
+
+Metadata length reports put the field and limit in the group heading; rows show
+only the measured length. Server timing's page list now includes only durations
+strictly greater than 50 ms. Aggregate timing statistics still include all
+responses, and `result.server_timing_summary` retains all timing data.
+
+### Redirect URLs are excluded from uniqueness checks
+
+Uniqueness checks now compare only HTML pages whose normalized requested and
+final URLs match. Redirect aliases no longer produce duplicate-title,
+duplicate-description, duplicate-text, or similar-text warnings from the
+destination's content. Redirect issues remain in the crawl report. Separate
+pages that serve identical content without a redirect are still checked.
+
+No host-app configuration changes are required. Applications that count
+uniqueness issues should expect fewer warnings when sitemap URLs redirect.
+
+### Clearer audit messages
+
+Text reports now use plain-language group headings with severity counts and
+show errors before warnings and notices. The repeated category summary is
+removed. Structured-data errors show the type, field, and constraint on separate
+lines (for example, `Article.headline: 125 characters; maximum 110`). Sitemap
+redirects show the destination and its final HTTP status once, without the host
+for same-origin URLs.
+Canonical warnings now identify the canonical target and explain the mismatch.
+Link warnings use `linked from` and `links to` to identify each URL's role.
+Length and ratio warnings include the limit and units. Reports show every issue
+and every related URL on separate lines. Duplicate-title and description values
+are shown in full. Link rules retain all source and target URLs rather than
+three samples. Missing-sitemap issues keep `source_url` and also provide
+`source_urls` with all pages that link to the missing URL. Server timing retains
+full descriptions and lists every page whose duration exceeds 50 ms.
+
+Host applications that parse report text or issue messages must update their
+matching patterns. Prefer issue codes and structured `details` for automation;
+these retain their existing fields, including the original schema errors,
+redirect status and final URL, and canonical href. No configuration changes are
+required.
+
+The former report limits `Reporter::MAX_ISSUES_PER_GROUP` and
+`Rules::Links::MAX_SOURCES_IN_ERROR` are removed. Host applications that reference
+these constants must remove those references.
+
+### Default sitemap is fetched over HTTP
+
+Crawlscope no longer prefers `public/sitemap.xml` when validating a localhost
+application. Without an explicit `SITEMAP` or configured `sitemap_path`, it now
+fetches `/sitemap.xml` from the configured base URL so the crawl observes the
+live application and database state.
+
+The Rails installer now generates the same HTTP default. Regenerate or update
+existing initializers that use `Rails.public_path.join("sitemap.xml")`.
+Explicit local sitemap paths remain supported through `SITEMAP`, `--sitemap`,
+or `config.sitemap_path`.
+
+`CRAWLSCOPE_PROFILE_TOKEN` is now the portable default for standalone CLI, Rake,
+Rails, and plain Ruby usage. Explicit `config.profile_token` values still take
+precedence.
+
+### Server-Timing report output
+
+No host configuration is required. When one or more responses publish a
+`Server-Timing` header, the text report now includes an optional `Server Timing`
+section. Host applications that parse report text should accept this additional
+section.
+
+Parsed metrics are available through `page.server_timing`, and aggregate data is
+available through `result.server_timing_summary`. Crawlscope interprets `dur`
+values as milliseconds and ignores malformed entries while reporting their
+count.
+
+### Ruby 3.3 is now required
+
+Crawlscope now depends on the current Async runtime for production async HTTP
+fetching. Host applications must run Ruby 3.3 or newer before upgrading.
+
+Recommended migration:
+
+1. Upgrade the host application runtime to Ruby 3.3 or newer.
+2. Run `bundle update crawlscope async async-http async-http-faraday`.
+3. Crawlscope now uses `FETCH_EXECUTOR=async` by default for HTTP crawling.
+4. Set `FETCH_EXECUTOR=threaded` or pass `--fetch-executor threaded` for a
+   conservative rollout or for explicit thread-pool execution.
+5. Browser rendering continues to use threaded execution by default because
+   async fetch execution is only supported with HTTP rendering.

data/lib/crawlscope/crawl.rb

@@ -114,7 +114,6 @@ module Crawlscope
         elsif !@allowed_statuses.include?(page.status)
           issues.add(code: :unexpected_status, severity: :error, category: :crawl, url: page.url, message: "HTTP #{page.status}", details: {status: page.status})
         elsif redirected?(page)
-          issues.add(code: :redirected_page, severity: :warning, category: :crawl, url: page.url, message: "redirects to #{page.final_url}", details: {final_url: page.final_url, status: page.status})
           issues.add(code: :sitemap_redirect_url, severity: :warning, category: :sitemaps, url: page.url, message: "sitemap URL redirects to #{page.final_url}", details: {final_url: page.final_url, status: page.status})
         end
       end

data/lib/crawlscope/rails.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+require "rails"
+require "rails/railtie"
+require "crawlscope"
+require "crawlscope/railtie"

data/lib/crawlscope/railtie.rb

@@ -1,5 +1,12 @@
 # frozen_string_literal: true
 
+require "rails"
+require "rails/railtie"
+
+unless Gem::Requirement.new(">= 8.0", "< 9").satisfied_by?(Gem.loaded_specs.fetch("railties").version)
+  raise LoadError, "crawlscope integration requires Rails 8.0 or newer within Rails 8"
+end
+
 module Crawlscope
   class Railtie < Rails::Railtie
     rake_tasks do