crawlscope 0.3.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b49aaaa6fdb5f7d5bd4dc63713d8c0090411e7063363645a900d8f59d803aaaa
-  data.tar.gz: 5dfcc35d60745c25db6faf3acaa4344e29e438c758740613d6216e2f47aeac6e
+  metadata.gz: 7d9e56ae9a55e3c4bb6b079585b8a302edcc1bfad9110938c9421c5224bf27f9
+  data.tar.gz: ab1908aa4a1bef4c15f055800ca9862ba973c9257f39020309de1f5554923cca
 SHA512:
-  metadata.gz: 9f66627274ce2ea969b5bb9b53a339215718c37baf47393c75bcf3a528c5c73658c6a71903fdbbf9e53796aaf3680be5f99ab4151b834efbf9450e05abbab83b
-  data.tar.gz: 3cf2e2c7f251a6af7b931f00da63436eaa7e09f078d73de112852a10665cf16eefb561c7d61d6bc8b0c3c014ca0db2df217d31c00b9f0ed321565ed554574261
+  metadata.gz: 8981de1e7bc19737df3048b1e19f28d585f22eed8f2b32ea4eea473ba377d3a261e08df8165114c8d967b7ec7d14a48c47a6b83cfe14261f6c83b56b39134766
+  data.tar.gz: 3df1e21bf74c12e994c0a932f9c581e4a6e12b55dd14975e43c5eca073bcc8eb4a49337d3c3c2e52137c4436fb5ddf52accd6fc11954171d77e75ce8f75e69a5

data/CHANGELOG.md

@@ -5,6 +5,35 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.5.0] - 2026-05-31
+
+
+### Added
+
+- expand SEO audit checks
+
+
+
+## [0.4.0] - 2026-05-21
+
+
+### Added
+
+- add indexability and content quality checks
+
+
+
+
+### Fixed
+
+- preserve release changelog history
+
+- scope content ratio to main content
+
+- harden indexability and uniqueness rules
+
+
+
 ## [0.3.0] - 2026-04-28
 
 
@@ -28,3 +57,47 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 
 
+## [0.2.0] - 2026-04-24
+
+
+### Changed
+
+- simplify crawl and structured data boundaries
+
+- harden validation boundaries
+
+
+
+
+### Fixed
+
+- handle child sitemaps
+
+- use URL for sitemap validation
+
+
+
+## [0.1.0] - 2026-04-23
+
+
+### Added
+
+- add crawlkit release-ready audit gem
+
+- add standalone validation commands
+
+- move default schema rules into crawlkit
+
+
+
+
+### Changed
+
+- strengthen public API coverage
+
+- load shared test dependencies
+
+- rename crawlkit to crawlscope
+
+
+

data/README.md

@@ -23,9 +23,11 @@ It works in three modes:
 
 The default rule set includes:
 
+- indexability blockers
 - metadata validation
 - structured-data validation
 - uniqueness checks
+- content-quality checks
 - internal-link checks
 
 ## Installation
@@ -146,9 +148,11 @@ Available tasks:
 
 ```bash
 bin/rails crawlscope:validate
+bin/rails crawlscope:validate:indexability
 bin/rails crawlscope:validate:metadata
 bin/rails crawlscope:validate:structured_data
 bin/rails crawlscope:validate:uniqueness
+bin/rails crawlscope:validate:content_quality
 bin/rails crawlscope:validate:links
 bin/rails crawlscope:validate:ldjson
 ```
@@ -157,11 +161,17 @@ The same validation surface is also available in the gem repository itself throu
 
 ```bash
 bundle exec rake crawlscope:validate URL=https://example.com
+bundle exec rake 'crawlscope:validate[https://example.com]'
 bundle exec rake crawlscope:validate:metadata URL=https://example.com
+bundle exec rake 'crawlscope:validate:metadata[https://example.com]'
 bundle exec rake crawlscope:validate:ldjson URL=https://example.com/article
+bundle exec rake 'crawlscope:validate:ldjson[https://example.com/article]'
 ```
 
-`crawlscope:validate` runs all default sitemap rules: metadata, structured data, uniqueness, and links. `URL` is the site base. Without `SITEMAP`, Crawlscope uses `/sitemap.xml`. With `SITEMAP`, Crawlscope uses `URL` as the site base and validates URLs from that sitemap. `SITEMAP` may be a full URL or a local file path.
+`crawlscope:validate` runs all default sitemap rules: indexability, metadata, structured data, uniqueness, content quality, and links. `URL` is the site base. Without `SITEMAP`, Crawlscope uses `/sitemap.xml`. With `SITEMAP`, Crawlscope uses `URL` as the site base and validates URLs from that sitemap. `SITEMAP` may be a full URL or a local file path.
+
+Plain `rake` does not pass `--url` style flags to tasks. Use `URL=...` or the
+task-argument form above instead.
 
 `crawlscope:validate:ldjson` is separate because it directly checks the URL or semicolon-separated URLs in `URL`; it does not crawl the sitemap. Without `URL`, it checks the configured base URL, falling back to `http://localhost:3000`.
 
@@ -186,11 +196,20 @@ Optional flags:
 
 Built-in rules:
 
+- `indexability`
 - `metadata`
 - `structured_data`
 - `uniqueness`
+- `content_quality`
 - `links`
 
+### Indexability
+
+Checks:
+
+- page-level meta robots `noindex`
+- `X-Robots-Tag: noindex`
+
 ### Metadata
 
 Checks:
@@ -220,6 +239,19 @@ Checks:
 - duplicate titles
 - duplicate meta descriptions
 - duplicate content fingerprints
+- near-duplicate visible content for up to 250 HTML pages
+
+For larger crawls, exact duplicate checks still run and Crawlscope reports
+`near_duplicate_scan_skipped`. Configure `Rules::Uniqueness` with
+`max_near_duplicate_pages:` in a custom rule registry to change the limit.
+
+### Content Quality
+
+Checks:
+
+- thin visible text
+- low visible-text-to-HTML ratio
+- low unique-token ratio
 
 ### Links
 
@@ -268,7 +300,12 @@ bundle exec rake
 
 ### Git hooks
 
-We use [lefthook](https://lefthook.dev/) with the Ruby [commitlint](https://github.com/arandilopez/commitlint) gem to enforce Conventional Commits on every commit. We also use [Standard Ruby](https://standardrb.com/) to keep code style consistent. CI validates commit messages, Standard Ruby, tests, and git-cliff changelog generation on pull requests and pushes to main/master.
+We use [lefthook](https://lefthook.dev/) with the Ruby
+[commitlint](https://github.com/arandilopez/commitlint) gem to enforce
+Conventional Commits on every commit. We also use
+[Standard Ruby](https://standardrb.com/) to keep code style consistent. CI
+validates commit messages, Standard Ruby, tests, and git-cliff changelog
+generation on pull requests and pushes to main/master.
 
 Run the hook installer once per clone:
 
@@ -284,11 +321,16 @@ rake install
 
 ## Release
 
-Releases are tag-driven and published by GitHub Actions to RubyGems. Local release commands never publish directly.
+Releases are tag-driven and published by GitHub Actions to RubyGems.
+Local release commands never publish directly.
 
-Install [git-cliff](https://git-cliff.org/) locally before preparing a release. The release task regenerates `CHANGELOG.md` from Conventional Commits.
+Install [git-cliff](https://git-cliff.org/) locally before preparing a
+release. The release task prepends the next `CHANGELOG.md` section from
+Conventional Commits.
 
-Before preparing a release, make sure you are on `main` or `master` with a clean worktree.
+Before preparing a release, make sure you are on `main` or `master` with a
+clean worktree. If the release contains a breaking public-contract change,
+update `UPGRADE.md` with the host-app migration steps first.
 
 Then run one of:
 
@@ -301,12 +343,13 @@ bundle exec rake 'release:prepare[0.1.0]'
 
 The task will:
 
-1. Regenerate `CHANGELOG.md` with `git-cliff`.
+1. Prepend the next `CHANGELOG.md` section with `git-cliff`.
 1. Update `lib/crawlscope/version.rb`.
 1. Commit the release changes.
 1. Create and push the `vX.Y.Z` tag.
 
-The `Release` workflow then runs tests, publishes the gem to RubyGems, and creates the GitHub release from the changelog entry.
+The `Release` workflow then runs tests, publishes the gem to RubyGems,
+and creates the GitHub release from the changelog entry.
 
 ## Contributing
 

data/lib/crawlscope/cli.rb

@@ -37,11 +37,14 @@ module Crawlscope
         @err.puts(general_usage)
         1
       end
-    rescue OptionParser::InvalidOption, OptionParser::MissingArgument, ConfigurationError, ValidationError, ArgumentError => error
+    rescue OptionParser::InvalidOption, OptionParser::MissingArgument, ConfigurationError, ArgumentError => error
       @err.puts(error.message)
       @err.puts("")
       @err.puts(general_usage)
       1
+    rescue ValidationError => error
+      @err.puts(error.message)
+      1
     end
 
     private

data/lib/crawlscope/crawl.rb

@@ -83,6 +83,7 @@ module Crawlscope
           issues.add(code: :unexpected_status, severity: :error, category: :crawl, url: page.url, message: "HTTP #{page.status}", details: {status: page.status})
         elsif redirected?(page)
           issues.add(code: :redirected_page, severity: :warning, category: :crawl, url: page.url, message: "redirects to #{page.final_url}", details: {final_url: page.final_url, status: page.status})
+          issues.add(code: :sitemap_redirect_url, severity: :warning, category: :sitemaps, url: page.url, message: "sitemap URL redirects to #{page.final_url}", details: {final_url: page.final_url, status: page.status})
         end
       end
     end
@@ -127,6 +128,7 @@ module Crawlscope
         crawled: crawled,
         error: page.error,
         final_url: page.normalized_final_url || normalized_url,
+        html: page.html?,
         status: page.status
       }
     end

data/lib/crawlscope/document_text.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module Crawlscope
+  module DocumentText
+    REMOVED_SELECTORS = "script, style, noscript, template, svg"
+    TOKEN_PATTERN = /[[:alnum:]]+/
+
+    module_function
+
+    def body_text(doc)
+      text_for(doc, selector: nil)
+    end
+
+    def html_for(doc, selector: "main")
+      root_for(doc, selector: selector)&.to_html.to_s
+    end
+
+    def text_for(doc, selector: "main")
+      normalize(root_for(doc, selector: selector)&.text)
+    end
+
+    def tokens(text)
+      normalize(text).downcase.scan(TOKEN_PATTERN).reject { |token| token.length < 2 }
+    end
+
+    def normalize(text)
+      text.to_s.gsub(/\s+/, " ").strip
+    end
+
+    def root_for(doc, selector:)
+      return unless doc
+
+      copy = doc.dup
+      copy.css(REMOVED_SELECTORS).remove
+
+      root = selector.to_s.empty? ? nil : copy.at_css(selector)
+      root || copy.at_css("body") || copy
+    end
+  end
+end

data/lib/crawlscope/rake_tasks.rb

@@ -4,25 +4,40 @@ module Crawlscope
   module RakeTasks
     module_function
 
-    def validate
-      run("validate")
+    def validate(url: nil, sitemap_path: nil, rule_names: nil)
+      run("validate", argv: validate_argv(url: url, sitemap_path: sitemap_path, rule_names: rule_names))
     end
 
-    def ldjson
-      run("ldjson")
+    def ldjson(urls: nil)
+      run("ldjson", argv: ldjson_argv(urls: urls))
     end
 
-    def validate_rule(rule)
-      original_rules = ENV["RULES"]
-      ENV["RULES"] = rule
-      validate
-    ensure
-      ENV["RULES"] = original_rules
+    def validate_rule(rule, url: nil, sitemap_path: nil)
+      validate(url: url, sitemap_path: sitemap_path, rule_names: rule)
     end
 
-    def run(command)
-      status = Cli.start([command], out: $stdout, err: $stderr)
+    def run(command, argv: [])
+      status = Cli.start([command, *argv], out: $stdout, err: $stderr)
       exit(status) unless status.zero?
     end
+
+    def validate_argv(url:, sitemap_path:, rule_names:)
+      [
+        option_pair("--url", url),
+        option_pair("--sitemap", sitemap_path),
+        option_pair("--rules", rule_names)
+      ].compact.flatten
+    end
+
+    def ldjson_argv(urls:)
+      Array(urls).flat_map { |url| option_pair("--url", url) }.compact
+    end
+
+    def option_pair(name, value)
+      value = value.to_s.strip
+      return if value.empty?
+
+      [name, value]
+    end
   end
 end

data/lib/crawlscope/reporter.rb

@@ -20,14 +20,29 @@ module Crawlscope
 
       @io.puts("Status: FAILED")
       @io.puts("Issues: #{result.issues.size}")
+      @io.puts("")
 
-      result.issues.by_severity.sort_by { |severity, _issues| severity.to_s }.each do |severity, issues|
-        @io.puts("#{severity}: #{issues.size}")
-      end
+      report_grouped_issues("Severity", result.issues.by_severity)
+      @io.puts("")
+      report_grouped_issues("Category", result.issues.by_category)
+    end
+
+    private
 
-      result.issues.each do |issue|
-        @io.puts("- [#{issue.severity}] #{issue.url} #{issue.message}")
+    def report_grouped_issues(title, grouped_issues)
+      @io.puts("#{title}:")
+
+      grouped_issues.sort_by { |name, _issues| name.to_s }.each do |name, issues|
+        @io.puts("#{name}: #{issues.size}")
+        issues.each do |issue|
+          @io.puts("  - #{offense(issue)}")
+        end
       end
     end
+
+    def offense(issue)
+      parts = ["[#{issue.severity}]", issue.code, issue.url, issue.message]
+      parts.compact.join(" ")
+    end
   end
 end

data/lib/crawlscope/rule_registry.rb

@@ -12,12 +12,14 @@ module Crawlscope
     def self.default(site_name: nil)
       new(
         rules: [
+          Rules::Indexability.new,
           Rules::Metadata.new(site_name: site_name),
           Rules::StructuredData.new,
           Rules::Uniqueness.new,
+          Rules::ContentQuality.new,
           Rules::Links.new
         ],
-        default_codes: %i[metadata structured_data uniqueness links]
+        default_codes: %i[indexability metadata structured_data uniqueness content_quality links]
       )
     end
 

data/lib/crawlscope/rules/content_quality.rb

@@ -0,0 +1,99 @@
+# frozen_string_literal: true
+
+module Crawlscope
+  module Rules
+    class ContentQuality
+      MIN_VISIBLE_TEXT_RATIO = 0.08
+      MIN_VISIBLE_WORDS = 250
+      MIN_UNIQUE_TOKEN_RATIO = 0.25
+
+      attr_reader :code
+
+      def initialize(
+        min_visible_text_ratio: MIN_VISIBLE_TEXT_RATIO,
+        min_visible_words: MIN_VISIBLE_WORDS,
+        min_unique_token_ratio: MIN_UNIQUE_TOKEN_RATIO
+      )
+        @code = :content_quality
+        @min_visible_text_ratio = min_visible_text_ratio
+        @min_visible_words = min_visible_words
+        @min_unique_token_ratio = min_unique_token_ratio
+      end
+
+      def call(urls:, pages:, issues:, context: nil)
+        pages.each do |page|
+          next unless page.html?
+
+          validate_visible_words(page, issues)
+          validate_visible_text_ratio(page, issues)
+          validate_unique_token_ratio(page, issues)
+        end
+      end
+
+      private
+
+      def validate_unique_token_ratio(page, issues)
+        tokens = DocumentText.tokens(DocumentText.text_for(page.doc))
+        return if tokens.size < @min_visible_words
+
+        ratio = tokens.uniq.size.to_f / tokens.size
+        return if ratio >= @min_unique_token_ratio
+
+        issues.add(
+          code: :low_unique_token_ratio,
+          severity: :warning,
+          category: :content_quality,
+          url: page.url,
+          message: "visible text has low token variety (#{format_ratio(ratio)})",
+          details: {
+            ratio: ratio.round(3),
+            threshold: @min_unique_token_ratio,
+            token_count: tokens.size,
+            unique_token_count: tokens.uniq.size
+          }
+        )
+      end
+
+      def validate_visible_text_ratio(page, issues)
+        html_bytes = DocumentText.html_for(page.doc).bytesize
+        return if html_bytes.zero?
+
+        visible_text = DocumentText.text_for(page.doc)
+        ratio = visible_text.bytesize.to_f / html_bytes
+        return if ratio >= @min_visible_text_ratio
+
+        issues.add(
+          code: :low_visible_text_ratio,
+          severity: :warning,
+          category: :content_quality,
+          url: page.url,
+          message: "low visible text to HTML ratio (#{format_ratio(ratio)})",
+          details: {
+            html_bytes: html_bytes,
+            ratio: ratio.round(3),
+            threshold: @min_visible_text_ratio,
+            visible_text_bytes: visible_text.bytesize
+          }
+        )
+      end
+
+      def validate_visible_words(page, issues)
+        word_count = DocumentText.tokens(DocumentText.text_for(page.doc)).size
+        return if word_count >= @min_visible_words
+
+        issues.add(
+          code: :thin_visible_text,
+          severity: :warning,
+          category: :content_quality,
+          url: page.url,
+          message: "thin visible text (#{word_count} words)",
+          details: {word_count: word_count, minimum: @min_visible_words}
+        )
+      end
+
+      def format_ratio(value)
+        format("%.2f", value)
+      end
+    end
+  end
+end

data/lib/crawlscope/rules/indexability.rb

@@ -0,0 +1,179 @@
+# frozen_string_literal: true
+
+module Crawlscope
+  module Rules
+    class Indexability
+      ROBOTS_META_SELECTOR = 'meta[name="robots"], meta[name="googlebot"]'
+      X_ROBOTS_TAG_HEADER = "x-robots-tag"
+
+      attr_reader :code
+
+      def initialize
+        @code = :indexability
+      end
+
+      def call(urls:, pages:, issues:, context: nil)
+        sitemap_urls = normalized_sitemap_urls(urls)
+
+        pages.each do |page|
+          validate_meta_robots(page, issues, sitemap_urls) if page.html?
+          validate_x_robots_tag(page, issues, sitemap_urls)
+        end
+      end
+
+      private
+
+      def normalized_sitemap_urls(urls)
+        urls.map { |url| Url.normalize(url, base_url: url) }.compact
+      end
+
+      def header_value(page, name)
+        page.headers.find { |key, _value| key.to_s.casecmp?(name) }&.last.to_s
+      end
+
+      def directives(value)
+        value
+          .split(",")
+          .map { |directive| directive.split(":", 2).last.to_s.strip }
+          .reject(&:empty?)
+      end
+
+      def noindex?(value)
+        directives(value).any? { |directive| directive.casecmp?("noindex") || directive.casecmp?("none") }
+      end
+
+      def follow?(value)
+        directives(value).any? { |directive| directive.casecmp?("follow") }
+      end
+
+      def nofollow?(value)
+        directives(value).any? { |directive| directive.casecmp?("nofollow") || directive.casecmp?("none") }
+      end
+
+      def validate_meta_robots(page, issues, sitemap_urls)
+        page.doc.css(ROBOTS_META_SELECTOR).each do |tag|
+          content = tag["content"].to_s
+
+          report_noindex_meta(page, issues, content, tag["name"].to_s, sitemap_urls) if noindex?(content)
+          report_nofollow_meta(page, issues, content, tag["name"].to_s) if nofollow?(content)
+          report_noindex_follow_meta(page, issues, content, tag["name"].to_s) if noindex?(content) && follow?(content)
+          report_noindex_nofollow_meta(page, issues, content, tag["name"].to_s) if noindex?(content) && nofollow?(content)
+        end
+      end
+
+      def validate_x_robots_tag(page, issues, sitemap_urls)
+        content = header_value(page, X_ROBOTS_TAG_HEADER)
+        return if content.empty?
+
+        report_noindex_header(page, issues, content, sitemap_urls) if noindex?(content)
+        report_nofollow_header(page, issues, content) if nofollow?(content)
+        report_noindex_follow_header(page, issues, content) if noindex?(content) && follow?(content)
+        report_noindex_nofollow_header(page, issues, content) if noindex?(content) && nofollow?(content)
+      end
+
+      def report_noindex_meta(page, issues, content, name, sitemap_urls)
+        issues.add(
+          code: :noindex_meta,
+          severity: :error,
+          category: :indexability,
+          url: page.url,
+          message: "robots meta tag prevents indexing",
+          details: {content: content, name: name}
+        )
+        report_sitemap_noindex_url(page, issues, content, source: "meta", sitemap_urls: sitemap_urls)
+      end
+
+      def report_nofollow_meta(page, issues, content, name)
+        issues.add(
+          code: :nofollow_meta,
+          severity: :warning,
+          category: :indexability,
+          url: page.url,
+          message: "robots meta tag prevents following links",
+          details: {content: content, name: name}
+        )
+      end
+
+      def report_noindex_follow_meta(page, issues, content, name)
+        issues.add(
+          code: :noindex_follow_meta,
+          severity: :warning,
+          category: :indexability,
+          url: page.url,
+          message: "robots meta tag prevents indexing but allows following links",
+          details: {content: content, name: name}
+        )
+      end
+
+      def report_noindex_nofollow_meta(page, issues, content, name)
+        issues.add(
+          code: :noindex_nofollow_meta,
+          severity: :error,
+          category: :indexability,
+          url: page.url,
+          message: "robots meta tag prevents indexing and following links",
+          details: {content: content, name: name}
+        )
+      end
+
+      def report_noindex_header(page, issues, content, sitemap_urls)
+        issues.add(
+          code: :noindex_header,
+          severity: :error,
+          category: :indexability,
+          url: page.url,
+          message: "X-Robots-Tag header prevents indexing",
+          details: {content: content}
+        )
+        report_sitemap_noindex_url(page, issues, content, source: "header", sitemap_urls: sitemap_urls)
+      end
+
+      def report_nofollow_header(page, issues, content)
+        issues.add(
+          code: :nofollow_header,
+          severity: :warning,
+          category: :indexability,
+          url: page.url,
+          message: "X-Robots-Tag header prevents following links",
+          details: {content: content}
+        )
+      end
+
+      def report_noindex_follow_header(page, issues, content)
+        issues.add(
+          code: :noindex_follow_header,
+          severity: :warning,
+          category: :indexability,
+          url: page.url,
+          message: "X-Robots-Tag header prevents indexing but allows following links",
+          details: {content: content}
+        )
+      end
+
+      def report_noindex_nofollow_header(page, issues, content)
+        issues.add(
+          code: :noindex_nofollow_header,
+          severity: :error,
+          category: :indexability,
+          url: page.url,
+          message: "X-Robots-Tag header prevents indexing and following links",
+          details: {content: content}
+        )
+      end
+
+      def report_sitemap_noindex_url(page, issues, content, source:, sitemap_urls:)
+        normalized_url = Url.normalize(page.url, base_url: page.url)
+        return unless sitemap_urls.include?(normalized_url)
+
+        issues.add(
+          code: :sitemap_noindex_url,
+          severity: :error,
+          category: :sitemaps,
+          url: page.url,
+          message: "sitemap URL is noindex",
+          details: {content: content, source: source}
+        )
+      end
+    end
+  end
+end