crawlscope 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 58a83d74a7b2b8422df4f161db9d3a7fe3ff213495f0837fd29a08cc13715b86
+  data.tar.gz: 02bd5743bcaae94bfdcc169fb6fe782257527984da68b091f6b75db3420b4244
+SHA512:
+  metadata.gz: c566f6899f45633db13a8ee47ac15f5e6054a4adff087774ce17ef15c26b10340694bd395e0de0efbdb5b652cf8ea04e3cbbb452d9467fd8167143f3675d5642
+  data.tar.gz: 1c087e1f4233224ea2c6b9b14de3bf34f4007b4689cd4fa8b9a3ea7ba688f78beb12431d0ffc7b6f54cae1eead319e3ba8293cef325440c614ca191b6ebf0e8b

data/CHANGELOG.md

@@ -0,0 +1,31 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2026-04-23
+
+
+### Added
+
+- add crawlkit release-ready audit gem
+
+- add standalone validation commands
+
+- move default schema rules into crawlkit
+
+
+
+
+### Changed
+
+- strengthen public API coverage
+
+- load shared test dependencies
+
+- rename crawlkit to crawlscope
+
+
+

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ethos Link
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,323 @@
+# Crawlscope
+
+[![Gem Version](https://badge.fury.io/rb/crawlscope.svg)](https://badge.fury.io/rb/crawlscope)
+[![Ruby](https://github.com/ethos-link/crawlscope/actions/workflows/ruby.yml/badge.svg)](https://github.com/ethos-link/crawlscope/actions/workflows/ruby.yml)
+
+`crawlscope` is a small Ruby gem for sitemap-driven SEO validation.
+
+It is built by [Ethos Link](https://www.ethos-link.com) and used in production by [Reviato](https://www.reviato.com).
+
+It is designed for Rails apps and plain Ruby scripts that want:
+
+- deterministic sitemap crawling
+- structured validation issues instead of free-form strings
+- app-configurable rule and schema registries
+- first-party rake tasks instead of a large DSL
+- optional browser rendering for JavaScript-heavy pages
+
+It works in three modes:
+
+- as a plain Ruby library
+- as a standalone CLI
+- as Rails rake tasks through the included Railtie
+
+The default rule set includes:
+
+- metadata validation
+- structured-data validation
+- uniqueness checks
+- internal-link checks
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem "crawlscope"
+```
+
+And then execute:
+
+```bash
+bundle install
+```
+
+Or install it directly:
+
+```bash
+gem install crawlscope
+```
+
+If you want browser rendering, also add:
+
+```ruby
+gem "ferrum"
+```
+
+`crawlscope` only loads Ferrum when you run in browser mode.
+
+## CLI Usage
+
+Validate a site directly from the gem:
+
+```bash
+crawlscope validate --base-url https://example.com
+```
+
+Validate only specific rules:
+
+```bash
+crawlscope validate --base-url https://example.com --rules metadata,links
+```
+
+Validate structured data on one or more URLs:
+
+```bash
+crawlscope ldjson --url https://example.com/article
+crawlscope ldjson --url https://example.com/a --url https://example.com/b --summary
+```
+
+If you do not pass `--sitemap`, `crawlscope` defaults to:
+
+- `https://example.com/sitemap.xml` for real site URLs
+- `public/sitemap.xml` for localhost-style development URLs when that file exists
+
+Child sitemap indexes are supported automatically.
+
+## Ruby Usage
+
+```ruby
+require "crawlscope"
+
+audit = Crawlscope::Audit.new(
+  base_url: "https://example.com",
+  sitemap_path: "https://example.com/sitemap.xml",
+  rules: Crawlscope::RuleRegistry.default(site_name: "Example").rules,
+  schema_registry: Crawlscope::SchemaRegistry.default
+)
+
+result = audit.call
+
+puts result.ok?
+puts result.issues.to_a.map(&:message)
+```
+
+## Result Shape
+
+`Crawlscope::Audit` returns a `Crawlscope::Result` with:
+
+- `urls`: sitemap URLs selected for validation
+- `pages`: fetched page snapshots
+- `issues`: structured issues with `code`, `severity`, `category`, `url`, and `message`
+
+`result.ok?` returns `false` if any error, warning, or notice is present.
+
+## Rails Usage
+
+In an initializer:
+
+```ruby
+Crawlscope.configure do |config|
+  config.base_url = -> { "https://example.com" }
+  config.sitemap_path = -> { Rails.public_path.join("sitemap.xml").to_s }
+  config.site_name = "Example"
+  config.schema_registry = -> { Crawlscope::SchemaRegistry.default }
+end
+```
+
+Then run:
+
+```bash
+bin/rails crawlscope:validate
+```
+
+Available environment overrides:
+
+- `BASE_URL`
+- `SITEMAP`
+- `RULES=metadata,links`
+- `JS=1` or `RENDERER=browser`
+- `TIMEOUT=30`
+- `NETWORK_IDLE_TIMEOUT=10`
+- `CONCURRENCY=5`
+
+Available tasks:
+
+```bash
+bin/rails crawlscope:validate
+bin/rails crawlscope:validate:metadata
+bin/rails crawlscope:validate:structured_data
+bin/rails crawlscope:validate:uniqueness
+bin/rails crawlscope:validate:links
+bin/rails crawlscope:validate:ldjson URL=https://example.com/article
+```
+
+The same validation surface is also available in the gem repository itself through plain `rake`:
+
+```bash
+bundle exec rake crawlscope:validate BASE_URL=https://example.com
+bundle exec rake crawlscope:validate:metadata BASE_URL=https://example.com
+bundle exec rake crawlscope:validate:ldjson URL=https://example.com/article
+```
+
+### Structured Data URL Audit
+
+For one-off structured-data checks:
+
+```bash
+bin/rails crawlscope:validate:ldjson URL=https://example.com/article
+bin/rails crawlscope:validate:ldjson URL='https://example.com/a;https://example.com/b' SUMMARY=1
+bin/rails crawlscope:validate:ldjson URL=https://example.com/article REPORT_PATH=tmp/structured-data.json
+```
+
+Optional flags:
+
+- `DEBUG=1`: print detected items
+- `SUMMARY=1`: print grouped failures
+- `REPORT_PATH=...`: write a JSON report
+- `JS=1` or `RENDERER=browser`: render with Ferrum
+
+## Rules
+
+Built-in rules:
+
+- `metadata`
+- `structured_data`
+- `uniqueness`
+- `links`
+
+### Metadata
+
+Checks:
+
+- missing `<h1>`
+- missing `<title>`
+- title length
+- repeated site name in the title
+- missing meta description
+- meta description length
+- missing canonical link
+- canonical mismatch
+
+### Structured Data
+
+Checks:
+
+- malformed JSON-LD
+- missing required fields for supported schema types
+- schema validation failures from the configured registry
+- direct URL structured-data audits through `crawlscope:validate:ldjson`
+
+### Uniqueness
+
+Checks:
+
+- duplicate titles
+- duplicate meta descriptions
+- duplicate content fingerprints
+
+### Links
+
+Checks:
+
+- broken internal links
+- unresolved internal links
+- low inbound anchor-link counts
+
+## Schema Registry
+
+`crawlscope` ships with a default schema registry for common types such as:
+
+- `Article`
+- `FAQPage`
+- `Organization`
+- `Product`
+- `Review`
+- `SoftwareApplication`
+- `WebApplication`
+- `WebSite`
+
+Host apps can replace or extend the registry:
+
+```ruby
+Crawlscope.configure do |config|
+  config.schema_registry = -> { MyApp::StructuredData::SchemaRegistry.new }
+end
+```
+
+That makes `crawlscope` useful as the audit engine while the app remains the owner of stricter product-specific schema rules.
+
+## Development
+
+```bash
+git clone https://github.com/ethos-link/crawlscope.git
+cd crawlscope
+
+bundle install
+bundle exec rake test
+bundle exec rake standard
+bundle exec rake
+```
+
+### Git hooks
+
+We use [lefthook](https://lefthook.dev/) with the Ruby [commitlint](https://github.com/arandilopez/commitlint) gem to enforce Conventional Commits on every commit. We also use [Standard Ruby](https://standardrb.com/) to keep code style consistent. CI validates commit messages, Standard Ruby, tests, and git-cliff changelog generation on pull requests and pushes to main/master.
+
+Run the hook installer once per clone:
+
+```bash
+bundle exec lefthook install
+```
+
+### Install locally
+
+```bash
+rake install
+```
+
+## Release
+
+Releases are tag-driven and published by GitHub Actions to RubyGems. Local release commands never publish directly.
+
+Install [git-cliff](https://git-cliff.org/) locally before preparing a release. The release task regenerates `CHANGELOG.md` from Conventional Commits.
+
+Before preparing a release, make sure you are on `main` or `master` with a clean worktree.
+
+Then run one of:
+
+```bash
+bundle exec rake 'release:prepare[patch]'
+bundle exec rake 'release:prepare[minor]'
+bundle exec rake 'release:prepare[major]'
+bundle exec rake 'release:prepare[0.1.0]'
+```
+
+The task will:
+
+1. Regenerate `CHANGELOG.md` with `git-cliff`.
+1. Update `lib/crawlscope/version.rb`.
+1. Commit the release changes.
+1. Create and push the `vX.Y.Z` tag.
+
+The `Release` workflow then runs tests, publishes the gem to RubyGems, and creates the GitHub release from the changelog entry.
+
+## Contributing
+
+1. Fork it
+1. Create a branch (`git checkout -b feature/my-feature`)
+1. Commit your changes
+1. Push (`git push origin feature/my-feature`)
+1. Open a Pull Request
+
+Please use [Conventional Commits](https://www.conventionalcommits.org/) for commit messages.
+
+## License
+
+MIT License, see [LICENSE.txt](LICENSE.txt)
+
+## About
+
+Made by the team at [Ethos Link](https://www.ethos-link.com) — practical software for growing businesses. We build tools for hospitality operators who need clear workflows, fast onboarding, and real human support.
+
+We also build [Reviato](https://www.reviato.com), “Capture. Interpret. Act.”.
+Turn guest feedback into clear next steps for your team. Collect private appraisals, spot patterns across reviews, and act before small issues turn into public ones.

data/exe/crawlscope

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "crawlscope"
+
+exit Crawlscope::Cli.start(ARGV)

data/lib/crawlscope/audit.rb

@@ -0,0 +1,128 @@
+# frozen_string_literal: true
+
+module Crawlscope
+  class Audit
+    def initialize(base_url:, sitemap_path:, rules:, schema_registry:, browser_factory: nil, concurrency: Configuration::DEFAULT_CONCURRENCY, network_idle_timeout_seconds: Configuration::DEFAULT_BROWSER_NETWORK_IDLE_TIMEOUT_SECONDS, renderer: :http, scroll_page: Configuration::DEFAULT_BROWSER_SCROLL_PAGE, timeout_seconds: Configuration::DEFAULT_TIMEOUT_SECONDS, allowed_statuses: Configuration::DEFAULT_ALLOWED_STATUSES)
+      @base_url = base_url
+      @sitemap_path = sitemap_path
+      @rules = Array(rules)
+      @schema_registry = schema_registry
+      @browser_factory = browser_factory
+      @concurrency = concurrency
+      @network_idle_timeout_seconds = network_idle_timeout_seconds
+      @renderer = renderer.to_sym
+      @scroll_page = scroll_page
+      @timeout_seconds = timeout_seconds
+      @allowed_statuses = allowed_statuses
+    end
+
+    def call
+      urls = Sitemap.new(path: @sitemap_path).urls(base_url: @base_url)
+      raise ValidationError, "No URLs found in sitemap: #{@sitemap_path}" if urls.empty?
+
+      @page_fetcher = build_page
+      pages = Crawler.new(
+        page_fetcher: @page_fetcher,
+        concurrency: @concurrency
+      ).call(urls)
+
+      issues = IssueCollection.new
+      collect_crawl_issues(pages, issues)
+      cache_pages(pages)
+      context = {
+        allowed_statuses: @allowed_statuses,
+        base_url: @base_url,
+        resolve_target: method(:resolve_target),
+        schema_registry: @schema_registry
+      }
+
+      @rules.each do |rule|
+        rule.call(urls: urls, pages: pages, issues: issues, context: context)
+      end
+
+      Result.new(
+        base_url: @base_url,
+        sitemap_path: @sitemap_path,
+        urls: urls,
+        pages: pages,
+        issues: issues
+      )
+    ensure
+      @page_fetcher&.close
+    end
+
+    private
+
+    def build_browser
+      Crawlscope::Browser.new(
+        base_url: @base_url,
+        timeout_seconds: @timeout_seconds,
+        network_idle_timeout_seconds: @network_idle_timeout_seconds,
+        scroll_page: @scroll_page
+      )
+    rescue LoadError => error
+      raise ConfigurationError, "Browser rendering requires the ferrum gem (#{error.message})"
+    end
+
+    def build_page
+      if @renderer == :browser
+        browser_factory = @browser_factory || method(:build_browser)
+        browser_factory.call
+      else
+        Http.new(base_url: @base_url, timeout_seconds: @timeout_seconds)
+      end
+    end
+
+    def build_target_resolution(page, normalized_target_url, crawled:)
+      {
+        crawled: crawled,
+        error: page.error,
+        final_url: page.normalized_final_url || normalized_target_url,
+        status: page.status
+      }
+    end
+
+    def cache_pages(pages)
+      @page_by_url = {}
+      @target_resolution_cache = {}
+
+      pages.each do |page|
+        @page_by_url[page.normalized_url] = page unless page.normalized_url.to_s.empty?
+        @page_by_url[page.normalized_final_url] = page unless page.normalized_final_url.to_s.empty?
+      end
+    end
+
+    def collect_crawl_issues(pages, issues)
+      pages.each do |page|
+        if page.error
+          issues.add(code: :fetch_failed, severity: :error, category: :crawl, url: page.url, message: page.error, details: {})
+        elsif !@allowed_statuses.include?(page.status)
+          issues.add(code: :unexpected_status, severity: :error, category: :crawl, url: page.url, message: "HTTP #{page.status}", details: {status: page.status})
+        end
+      end
+    end
+
+    def resolve_target(target_url)
+      normalized_target_url = Url.normalize(target_url, base_url: @base_url)
+      return @target_resolution_cache[normalized_target_url] if @target_resolution_cache.key?(normalized_target_url)
+
+      resolution = resolve_from_crawled_page(normalized_target_url)
+      resolution ||= resolve_by_fetching_target(normalized_target_url)
+      @target_resolution_cache[normalized_target_url] = resolution
+    end
+
+    def resolve_by_fetching_target(normalized_target_url)
+      page = @page_fetcher.fetch(normalized_target_url)
+      @page_by_url[page.normalized_url] = page unless page.normalized_url.to_s.empty?
+      @page_by_url[page.normalized_final_url] = page unless page.normalized_final_url.to_s.empty?
+      build_target_resolution(page, normalized_target_url, crawled: false)
+    end
+
+    def resolve_from_crawled_page(normalized_target_url)
+      page = @page_by_url[normalized_target_url]
+      return if page.nil?
+
+      build_target_resolution(page, normalized_target_url, crawled: true)
+    end
+  end
+end

data/lib/crawlscope/browser.rb

@@ -0,0 +1,88 @@
+# frozen_string_literal: true
+
+require "nokogiri"
+
+module Crawlscope
+  class Browser
+    def initialize(base_url:, timeout_seconds:, network_idle_timeout_seconds:, scroll_page:)
+      @base_url = base_url
+      @timeout_seconds = timeout_seconds
+      @network_idle_timeout_seconds = network_idle_timeout_seconds
+      @scroll_page = scroll_page
+      @browser = build_browser
+      @page = @browser.create_page
+    end
+
+    def close
+      @browser&.quit
+    end
+
+    def fetch(url)
+      @page.network.clear(:traffic)
+      @page.go_to(url)
+      wait_for_network_idle
+
+      if @scroll_page
+        scroll_for_render
+      end
+
+      response = @page.network.response
+      final_url = response&.url.to_s
+      final_url = @page.current_url.to_s if final_url.empty?
+      final_url = @page.url.to_s if final_url.empty?
+      final_url = url if final_url.empty?
+      headers = response&.headers || {}
+      body = @page.body
+
+      Page.new(
+        url: url,
+        normalized_url: Url.normalize(url, base_url: @base_url),
+        final_url: final_url,
+        normalized_final_url: Url.normalize(final_url, base_url: @base_url),
+        status: @page.network.status,
+        headers: headers,
+        body: body,
+        doc: Nokogiri::HTML(body)
+      )
+    rescue => error
+      Page.new(
+        url: url,
+        normalized_url: Url.normalize(url, base_url: @base_url),
+        final_url: url,
+        normalized_final_url: Url.normalize(url, base_url: @base_url),
+        status: nil,
+        headers: {},
+        body: nil,
+        doc: nil,
+        error: "#{error.class}: #{error.message}"
+      )
+    end
+
+    private
+
+    def build_browser
+      require "ferrum"
+
+      Ferrum::Browser.new(
+        headless: true,
+        timeout: @timeout_seconds,
+        headers: {"User-Agent" => Http::USER_AGENT}
+      )
+    end
+
+    def scroll_for_render
+      @page.evaluate("(function() { if (document.body) { window.scrollTo(0, document.body.scrollHeight); } })()")
+      wait_for_network_idle
+      @page.evaluate("(function() { if (document.body) { window.scrollTo(0, 0); } })()")
+      wait_for_network_idle
+      @page.evaluate("(function() { if (document.body) { window.scrollTo(0, document.body.scrollHeight / 2); } })()")
+      wait_for_network_idle
+    end
+
+    def wait_for_network_idle
+      @page.network.wait_for_idle(duration: 0.5, timeout: @network_idle_timeout_seconds)
+    rescue Ferrum::TimeoutError
+      raise Timeout::Error, "Timed out waiting for browser network idle"
+    end
+  end
+end