seo_geo 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ec8b0f1b5e2f745e00c47a2d2f9a15e2eda6ca9d0357948feca1a22548c6ec25
-  data.tar.gz: 8eb32a70fd40779db958d83a6927229c7b09ba156e630472cb4325759cdbec91
+  metadata.gz: bcf366096f21a5775c68799a9d2daf6a221d228ce0537c032050993e6703ea52
+  data.tar.gz: 9917d75bca5dae4ddf96f9d99567361d5fb7c66a0ec6048faabdf5ae1e24964f
 SHA512:
-  metadata.gz: 01636e3e7a699db78db0765973b16652cd6130c2ed1aea1c463632c3b8b55cd1576bdbedfcbf20c0f0f95139ee64c9579a735d3129bc7af1815960a94881389d
-  data.tar.gz: 33f4593adfa66ddebdf586bdc21473749eb60df802d242da5c2b8be2fc983a48af5338da4ab55f65b46272768070e02cafeb870cf3c896888fd30120d4ac7c63
+  metadata.gz: '09ce397eac4ae10fbabde19a7ef53335a7a09e9e9e1756c56f7d4e57834c5ca9947cc315966c5bbe11d2b5c70db2810afa262d0c2973ab9541e17f56fa1d4272'
+  data.tar.gz: 7f49703ed03915d08bdee4c5155c115aba8e8c9038312a61773bd15a77374da1d0f788a617123b46418c046e1641f1d18f291c6762d922ff02b0a815c24676f2

data/AGENTS.md

@@ -162,6 +162,29 @@ end
 RSpec: `require "seo_geo/rspec"` gives `have_seo_geo_canonical`, `have_seo_geo_single_title`,
 `have_seo_geo_faq_matching_visible_text`, `be_seo_geo_valid_sitemap(origin:)` and `seo_geo_robots_allow(ua, path)`.
 
+For article pages, pass `type: "article"`, `published_time:` and `modified_time:` to `seo`.
+Use the article's real stored dates: `Time`, `Date` and ISO 8601 strings are accepted and output as ISO 8601.
+Date-only values remain date-only; invalid values raise `ArgumentError` when rendering an article.
+These tags are omitted for other OG types and when no time is supplied.
+
+<!-- example: quickstart/app/controllers/articles_controller.rb -->
+```ruby
+# app/controllers/articles_controller.rb
+class ArticlesController < ApplicationController
+  def show
+    seo title: "Example launch", type: "article",
+        published_time: Time.iso8601("2026-10-01T09:00:00+09:00"),
+        modified_time: "2026-10-02T12:00:00+09:00"
+  end
+end
+```
+
+<!-- output: quickstart GET /articles/launch Host=www.example.com -->
+```html
+<meta property="article:published_time" content="2026-10-01T09:00:00+09:00">
+<meta property="article:modified_time" content="2026-10-02T12:00:00+09:00">
+```
+
 ## Invariants — do not violate
 
 1. **Hosts fail closed.** Every hostname that serves the app is either `canonical_origin`'s host, listed in
@@ -197,6 +220,25 @@ The rake task audits as production even in development: it sets `SEO_GEO_AUDIT=1
 the configured hosts to `config.hosts` for that run (`seo_geo audit: temporarily allowing ... in config.hosts`).
 A runner needs the variable before boot. Production `config.hosts` is never widened.
 
+Audit deployed HTTP responses without booting the application with `URL` or comma-separated `URLS`:
+
+```sh
+bundle exec rake seo_geo:audit URL=https://www.example.com
+bundle exec rake seo_geo:audit URLS=https://www.example.com,https://www.example.com/pricing FORMAT=json
+```
+
+Remote requests run serially with a 0.5-second pause between requests, 10-second connect/read/write timeouts
+and `User-Agent: seo_geo-audit/<VERSION>`. The initial response remains the audit input; redirect destinations
+are explicitly probed and recorded up to five hops. More than one hop warns (`page.redirect_chain`), while loops,
+unfinished chains and `X-Robots-Tag: noindex` (including scoped `none`) are errors. `robots.mismatch` warns only
+when an already available configuration declares the host and its rendered robots.txt differs from the response.
+No Rails initializer is loaded for remote mode, so pass an existing configuration via
+`SeoGeo::Audit::Remote.run(urls, io: $stdout, config: config)` when a comparison is needed.
+Without configuration, canonical URLs are checked against the requested origin and robots blocking against `*`;
+intentional crawler-specific denials cannot be inferred. Both audit modes probe a random missing path per host
+(`page.soft_404` on 2xx) and warn about duplicate descriptions on indexable pages (`description.duplicate`).
+Remote audits judge the live site as production would: like the in-process audit they switch the config to an indexable environment for the run and restore it afterwards. They do not widen `config.hosts` or scan local source files.
+
 With the app running (`bin/rails s`), check what crawlers and agents receive. robots.txt and sitemap.xml only
 open up in `indexable_environments` (default production):
 

data/CHANGELOG.md

@@ -1,6 +1,12 @@
 # Changelog
 
-## Unreleased
+## 0.2.0 - 2026-10-20
+
+- Add standalone remote audits (`URL` / `URLS`, `Audit::Remote.run`) with shared checks, Rack/HTTP fetchers,
+  serial paced requests, bounded redirect tracing, X-Robots-Tag errors and configured robots comparison.
+- Detect soft-404 responses with a random missing path on each audited host, in-process and remotely.
+- Warn when indexable pages on the same host share a meta description.
+- Support `seo published_time:` / `modified_time:` as ISO 8601 article Open Graph tags, with executable examples.
 
 ## 0.1.0 - 2026-10-14
 

data/README.md

@@ -61,8 +61,7 @@ gem "seo_geo"
 
 ## Status
 
-- Version 0.1.0, **not yet released** to RubyGems. Until then, use
-  `gem "seo_geo", github: "seunghan91/seo_geo"`.
+- Version 0.2.0, released to [RubyGems](https://rubygems.org/gems/seo_geo) on 2026-10-20.
 - CI runs Ruby 3.2, 3.3 and 3.4 × Rails 7.2, 8.0 and 8.1 (every cell), each against an ERB app and an
   Inertia-style app.
 - Runtime dependencies: `railties`, `actionpack`, `meta-tags ~> 2.24` and `reverse_markdown ~> 3.0`.
@@ -109,6 +108,7 @@ SeoGeo.config.llms_txt.page :home, path: "/"
 Rails.application.routes.draw do
   root "pages#home"
   get "/pricing", to: "pages#pricing"
+  get "/articles/launch", to: "articles#show"
 
   mount SeoGeo::Engine, at: "/"  # last, so your own routes win where paths overlap
 end
@@ -276,6 +276,48 @@ On a real app warnings appear in the same line, for example
 line per finding. Calling `SeoGeo::Audit::Runner` from `bin/rails runner` needs `SEO_GEO_AUDIT=1` set before
 boot for the same host allowance.
 
+Audit deployed HTTP responses without booting the application with `URL` or comma-separated `URLS`:
+
+```sh
+bundle exec rake seo_geo:audit URL=https://www.example.com
+bundle exec rake seo_geo:audit URLS=https://www.example.com,https://www.example.com/pricing FORMAT=json
+```
+
+Remote requests run serially with a 0.5-second pause between requests, 10-second connect/read/write timeouts
+and `User-Agent: seo_geo-audit/<VERSION>`. The initial response remains the audit input; redirect destinations
+are explicitly probed and recorded up to five hops. More than one hop warns (`page.redirect_chain`), while loops,
+unfinished chains and `X-Robots-Tag: noindex` (including scoped `none`) are errors. `robots.mismatch` warns only
+when an already available configuration declares the host and its rendered robots.txt differs from the response.
+No Rails initializer is loaded for remote mode, so pass an existing configuration via
+`SeoGeo::Audit::Remote.run(urls, io: $stdout, config: config)` when a comparison is needed.
+Without configuration, canonical URLs are checked against the requested origin and robots blocking against `*`;
+intentional crawler-specific denials cannot be inferred. Both audit modes probe a random missing path per host
+(`page.soft_404` on 2xx) and warn about duplicate descriptions on indexable pages (`description.duplicate`).
+Remote audits judge the live site as production would: like the in-process audit they switch the config to an indexable environment for the run and restore it afterwards. They do not widen `config.hosts` or scan local source files.
+
+For article pages, pass `type: "article"`, `published_time:` and `modified_time:` to `seo`.
+Use the article's real stored dates: `Time`, `Date` and ISO 8601 strings are accepted and output as ISO 8601.
+Date-only values remain date-only; invalid values raise `ArgumentError` when rendering an article.
+These tags are omitted for other OG types and when no time is supplied.
+
+<!-- example: quickstart/app/controllers/articles_controller.rb -->
+```ruby
+# app/controllers/articles_controller.rb
+class ArticlesController < ApplicationController
+  def show
+    seo title: "Example launch", type: "article",
+        published_time: Time.iso8601("2026-10-01T09:00:00+09:00"),
+        modified_time: "2026-10-02T12:00:00+09:00"
+  end
+end
+```
+
+<!-- output: quickstart GET /articles/launch Host=www.example.com -->
+```html
+<meta property="article:published_time" content="2026-10-01T09:00:00+09:00">
+<meta property="article:modified_time" content="2026-10-02T12:00:00+09:00">
+```
+
 ## Inertia (React, Vue, Svelte)
 
 Inertia sends an empty `<div id="app">`, and crawlers that do not run JavaScript (GPTBot, ClaudeBot,
@@ -396,7 +438,7 @@ are read with `find_each` (`select` must include `id`, `order` is ignored); `bat
 | `title_template must contain %{title}` | A template such as `"%{site}"` | Include `%{title}`, or use `"%{title}"` to drop the suffix |
 | `CANONICAL_ORIGIN must look like https://www.example.com (got "example.com")` | Generator argument without a scheme | `bin/rails g seo_geo:install https://www.example.com` |
 | `SeoGeo::JsonLd::InvalidNode: Product "Widget" needs offers, review or aggregate_rating` | A typed JSON-LD node lacks a property Google requires | Pass the property, or leave the node out; never invent one |
-| `unknown seo option(s): titel (known: title, description, image, noindex, index, canonical, page, type, markdown, json_ld)` | Typo in a controller `seo(...)` call | Use one of the listed keys |
+| `unknown seo option(s): titel (known: title, description, image, noindex, index, canonical, page, type, markdown, json_ld, published_time, modified_time)` | Typo in a controller `seo(...)` call | Use one of the listed keys |
 | `KeyError: seo_geo page :pricing is not defined` | No `config/seo_geo/pages/pricing.yml`, I18n scope or `SeoGeo.pages.register` | Add the page copy, or check the key and the YAML locale keys |
 | `collections need path: (a callable or a record method name)` | `sitemap.add -> { Post.published }` without `path:` | `add -> { Post.published }, path: ->(post) { "/posts/#{post.slug}" }` |
 | `sitemap path "pricing" must start with /` | Relative sitemap path | `add "/pricing"` |

data/lib/seo_geo/audit/http_fetcher.rb

@@ -0,0 +1,99 @@
+# frozen_string_literal: true
+
+require "net/http"
+require "uri"
+require "thread"
+require "rack/mock"
+
+module SeoGeo
+  module Audit
+    # Serial HTTP transport. The initial response is preserved; redirect targets are separate probes.
+    class HttpFetcher
+      REDIRECTS = [301, 302, 303, 307, 308].freeze
+      MAX_HOPS = 5
+      # @return [Array<Hash>] initial response and each explicitly requested redirect target
+      attr_reader :chain
+      # @return [String, nil] why the redirect trace could not finish
+      attr_reader :redirect_error
+
+      # @param origin [String] HTTP(S) origin to request
+      # @param delay [Numeric] minimum pause between requests, including redirect probes
+      # @param timeout [Numeric] connect/read/write timeout in seconds
+      def initialize(origin:, delay: 0.5, timeout: 10)
+        @origin = self.class.http_uri(origin).to_s
+        raise ArgumentError, "delay must be nonnegative and timeout positive" unless delay >= 0 && timeout > 0
+
+        @delay = delay
+        @timeout = timeout
+        @mutex = Mutex.new
+        @chain = []
+      end
+
+      # Validates caller URLs and redirect locations before opening a connection.
+      # @return [URI::HTTP]
+      def self.http_uri(url)
+        uri = URI.parse(url.to_s)
+        unless uri.is_a?(URI::HTTP) && uri.host && !uri.userinfo
+          raise ArgumentError, "audit URL must be HTTP(S), with a host and without credentials"
+        end
+        uri.path = "/" if uri.path.empty?
+        uri.fragment = nil
+        uri
+      end
+
+      # @return [Rack::MockResponse] first response (never replaced by a redirect target)
+      def fetch(_host, path, headers = {})
+        @mutex.synchronize do
+          @chain = []
+          @redirect_error = nil
+          uri = self.class.http_uri(URI.join(@origin, path).to_s)
+          first = nil
+          loop do
+            response = request(uri, headers)
+            first ||= response
+            @chain << { url: uri.to_s, status: response.status, headers: response.headers }
+            break unless REDIRECTS.include?(response.status) && response.headers["location"]
+            if @chain.size > MAX_HOPS
+              @redirect_error = "redirect chain exceeds #{MAX_HOPS} hops"
+              break
+            end
+            target = self.class.http_uri(URI.join(uri.to_s, response.headers["location"]).to_s)
+            if @chain.any? { |hop| hop[:url] == target.to_s }
+              @redirect_error = "redirect loop to #{target}"
+              break
+            end
+            uri = target
+          end
+          first
+        rescue StandardError => e
+          raise unless first
+
+          @redirect_error = "#{e.class}: #{e.message.lines.first.to_s.strip}"
+          first
+        end
+      end
+
+      private
+
+      def request(uri, headers)
+        if @last_request
+          remaining = @delay - (Process.clock_gettime(Process::CLOCK_MONOTONIC) - @last_request)
+          sleep(remaining) if remaining.positive?
+        end
+        request = Net::HTTP::Get.new(uri.request_uri)
+        request["Accept"] = "text/html"
+        headers.each { |name, value| request[name.delete_prefix("HTTP_").tr("_", "-")] = value }
+        request["User-Agent"] = "seo_geo-audit/#{SeoGeo::VERSION}"
+        # Ignore ambient proxy credentials/configuration; audit the requested origin directly.
+        http = Net::HTTP.new(uri.hostname, uri.port, nil)
+        http.use_ssl = uri.scheme == "https"
+        http.open_timeout = http.read_timeout = http.write_timeout = @timeout
+        http.max_retries = 0
+        response = http.start { |connection| connection.request(request) }
+        Rack::MockResponse.new(response.code.to_i, response.to_hash.transform_values { |values| values.join("\n") }, response.body.to_s)
+      ensure
+        @last_request = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      end
+    end
+  end
+end

data/lib/seo_geo/audit/rack_fetcher.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+require "rack"
+require "rack/mock"
+require "uri"
+
+module SeoGeo
+  module Audit
+    # The audit's original in-process Rack transport.
+    class RackFetcher
+      # @param app [#call] Rack application
+      # @param config [Configuration] origins for the audited hosts
+      def initialize(app:, config: SeoGeo.config)
+        @app = app
+        @config = config
+      end
+
+      # @return [Rack::MockResponse] original response, without following redirects
+      def fetch(host, path, headers = {})
+        origin = URI.parse(@config.origin_for(host))
+        scheme = origin.scheme || "https"
+        # Rails HostAuthorization (config.hosts) reads the Host header, not SERVER_NAME: send both, with the port
+        # only when it is not the scheme's default.
+        authority = origin.port && origin.port != origin.default_port ? "#{host}:#{origin.port}" : host
+        env = Rack::MockRequest.env_for("#{scheme}://#{authority}#{path}",
+                                        { "REMOTE_ADDR" => "127.0.0.1", "HTTP_ACCEPT" => "text/html" }.merge(headers))
+        env["HTTP_HOST"] = authority
+        env["HTTPS"] = "on" if scheme == "https"
+        status, response_headers, body = @app.call(env)
+        Rack::MockResponse.new(status, response_headers, body)
+      end
+    end
+  end
+end

data/lib/seo_geo/audit/remote.rb

@@ -0,0 +1,114 @@
+# frozen_string_literal: true
+
+module SeoGeo
+  module Audit
+    # Audits deployed responses without booting Rails or changing the application's configuration.
+    class Remote < Runner
+      # @param urls [Array<String>, String] explicit pages/origins; each origin is audited once
+      # @param io [#puts, nil] report destination; nil suppresses output
+      # @param config [Configuration, nil] optional comparison policy, used only for known hosts
+      # @param max_pages [Integer] sitemap pages per origin
+      # @param delay [Numeric] HTTP request spacing in seconds
+      # @param timeout [Numeric] HTTP timeouts in seconds
+      # @param format [Symbol, String] text or json
+      # @return [Report]
+      def self.run(urls, io: $stdout, config: SeoGeo.config, max_pages: 200, delay: 0.5, timeout: 10, format: :text)
+        targets = Array(urls).map { |url| HttpFetcher.http_uri(url) }
+        raise ArgumentError, "at least one audit URL is required" if targets.empty?
+
+        transport = HttpFetcher.new(origin: targets.first.to_s, delay: delay, timeout: timeout)
+        report = Report.new
+        targets.group_by { |uri| "#{uri.scheme}://#{uri.authority}" }.each do |origin, pages|
+          result = new(origin: origin, urls: pages.map(&:to_s), config: config,
+                       max_pages: max_pages, delay: delay, timeout: timeout, fetcher: transport).run
+          report.findings.concat(result.findings)
+          result.stats.each { |name, count| report.stats[name] += count }
+        end
+        io.puts(format.to_s.casecmp?("json") ? report.to_json : report.to_s) if io
+        report
+      end
+
+      # @param origin [String] one HTTP(S) origin
+      # @param urls [Array<String>] explicitly requested pages on that origin
+      # @param config [Configuration, nil] optional existing configuration
+      # @param max_pages [Integer] maximum sitemap pages
+      # @param delay [Numeric] delay between HTTP requests
+      # @param timeout [Numeric] HTTP timeouts
+      # @param fetcher [HttpFetcher, nil] shared serial transport across origins
+      def initialize(origin:, urls:, config: nil, max_pages: 200, delay: 0.5, timeout: 10, fetcher: nil)
+        @origin = origin
+        @urls = urls.uniq
+        host = URI.parse(origin).host.downcase
+        @comparison_config = config if config&.known_hosts&.include?(host)
+        observed = @comparison_config || Configuration.new.tap do |settings|
+          settings.canonical_origin = origin
+          settings.environment = "production"
+        end
+        @audited_pages = []
+        @remote_findings = {}
+        super(app: nil, config: observed, sitemap: Sitemap::Registry.new, hosts: [host], max_pages: max_pages,
+              fetcher: fetcher || HttpFetcher.new(origin: origin, delay: delay, timeout: timeout))
+      end
+
+      private
+
+      def origin_for(_host)
+        @origin
+      end
+
+      def audit_host(host)
+        super
+        @urls.each do |url|
+          audit_page(host, url, from_sitemap: false) unless @audited_pages.include?(url)
+        end
+      end
+
+      def audit_page(host, url, **options)
+        @audited_pages << url
+        super
+      end
+
+      def sitemap_urls(host)
+        super.select { |url| Checks.same_origin?(url, @origin) }
+      end
+
+      def check_robots_allows(host, url, evaluator)
+        return super if @comparison_config
+        return if evaluator.allowed?("*", request_path(url))
+
+        error(host, url, "sitemap.robots_blocked", "listed in sitemap.xml but robots.txt disallows it for *")
+      end
+
+      def fetch(host, path, headers = {}, **options)
+        response = super(host, "#{@origin}#{path}", headers, **options)
+        @fetcher.chain.each do |hop|
+          directives = RobotsDirectives.parse_header(hop[:headers]["x-robots-tag"])
+          if directives.noindex? || directives.scoped.keys.any? { |bot| directives.noindex?(bot) }
+            remote_finding(:error, host, hop[:url], "page.x_robots_noindex", "X-Robots-Tag contains noindex")
+          end
+        end
+        hops = @fetcher.chain.size - 1
+        if hops > 1
+          trace = @fetcher.chain.map { |hop| "#{hop[:status]} #{hop[:url]}" }.join(" -> ")
+          remote_finding(:warning, host, path, "page.redirect_chain", "#{hops} redirect hops: #{trace}")
+        end
+        if @fetcher.redirect_error
+          remote_finding(:error, host, path, "page.redirect", @fetcher.redirect_error)
+        end
+        if path == "/robots.txt" && @comparison_config && response.status == 200 &&
+           response.body != Robots.new(@comparison_config, host).to_s
+          remote_finding(:warning, host, path, "robots.mismatch", "robots.txt differs from the configured rendering for this host")
+        end
+        response
+      end
+
+      def remote_finding(severity, host, url, check, message)
+        key = [host, url, check]
+        return if @remote_findings[key]
+
+        @remote_findings[key] = true
+        @report.add(severity, check, message, host: host, url: url)
+      end
+    end
+  end
+end

data/lib/seo_geo/audit/runner.rb

@@ -3,6 +3,7 @@
 require "uri"
 require "rack"
 require "rack/mock"
+require "securerandom"
 
 module SeoGeo
   module Audit
@@ -29,14 +30,17 @@ module SeoGeo
       # @param root [String, Pathname, nil] application root for {StaticScan}; nil skips it
       # @param hosts [Array<String>, nil] hosts to audit; nil => {Configuration#known_hosts}
       # @param max_pages [Integer] sitemap URLs fetched per host
-      def initialize(app:, config: SeoGeo.config, sitemap: SeoGeo.sitemap, root: nil, hosts: nil, max_pages: 200)
+      # @param fetcher [#fetch, nil] transport; nil uses {RackFetcher}
+      def initialize(app:, config: SeoGeo.config, sitemap: SeoGeo.sitemap, root: nil, hosts: nil, max_pages: 200, fetcher: nil)
         @app = app
+        @fetcher = fetcher || RackFetcher.new(app: app, config: config)
         @config = config
         @sitemap = sitemap
         @root = root
         @hosts = hosts
         @max_pages = max_pages
         @report = Report.new
+        @descriptions = Hash.new { |hash, key| hash[key] = [] }
         @titles = Hash.new { |hash, key| hash[key] = [] }
       end
 
@@ -48,12 +52,14 @@ module SeoGeo
           (@hosts || @config.known_hosts).each { |host| audit_host(host.to_s.downcase) }
         end
         duplicate_titles
+        duplicate_descriptions
         @report
       end
 
       private
 
       def audit_host(host)
+        check_soft_404(host)
         policy = @config.host_policy(host)
         robots = fetch(host, "/robots.txt")
         evaluator = nil
@@ -70,7 +76,7 @@ module SeoGeo
           check_robots_allows(host, url, evaluator) if evaluator
           audit_page(host, url, from_sitemap: true)
         end
-        home = "#{@config.origin_for(host)}/"
+        home = "#{origin_for(host)}/"
         unless pages.any? { |url| same_page?(url, home) }
           audit_page(host, home, from_sitemap: false, full: policy.sitemap)
         end
@@ -84,7 +90,7 @@ module SeoGeo
           return []
         end
 
-        origin = "#{@config.origin_for(host)}/"
+        origin = "#{origin_for(host)}/"
         problems, locs, kind = Checks.sitemap(response.body, origin: origin)
         problems.each { |message| error(host, "/sitemap.xml", "sitemap.valid", message) }
         return locs unless kind == :sitemapindex
@@ -155,6 +161,7 @@ module SeoGeo
           @titles[[host, title]] << url unless noindex
         end
         description = doc.description.to_s.strip
+        @descriptions[[host, description]] << url unless noindex || description.empty?
         if description.empty?
           warning(host, url, "description.missing", "no meta description")
         elsif !Checks::DESCRIPTION_RANGE.cover?(description.length)
@@ -183,7 +190,7 @@ module SeoGeo
           error(host, markdown_url, "markdown.url", "advertised Markdown URL returned #{dot_md.status} #{dot_md.content_type}")
         end
 
-        negotiated = fetch(host, request_path(url), "HTTP_ACCEPT" => "text/markdown")
+        negotiated = fetch(host, request_path(url), { "HTTP_ACCEPT" => "text/markdown" })
         unless negotiated.status == 200 && negotiated.content_type.to_s.match?(MARKDOWN_TYPE)
           error(host, url, "markdown.accept", "page advertises Markdown but Accept: text/markdown returned " \
                                               "#{negotiated.status} #{negotiated.content_type} (include SeoGeo::Markdownable " \
@@ -201,7 +208,7 @@ module SeoGeo
           return
         end
 
-        origins = [@config.origin_for(host), @config.canonical_origin].uniq
+        origins = [origin_for(host), @config.canonical_origin].uniq
         response.body.scan(/\]\((https?:\/\/[^)\s]+)\)/).flatten.uniq.each do |link|
           next unless origins.any? { |origin| link.start_with?("#{origin}/") || link == origin }
 
@@ -230,27 +237,42 @@ module SeoGeo
         end
       end
 
+      def duplicate_descriptions
+        @descriptions.each do |(host, description), urls|
+          next if urls.uniq.size < 2
+
+          warning(host, urls.first, "description.duplicate",
+                  "description #{description.inspect} is shared by #{urls.uniq.size} URLs: #{urls.uniq.first(5).join(', ')}")
+        end
+      end
+
+      def check_soft_404(host)
+        path = "/seo-geo-audit-#{SecureRandom.hex(8)}"
+        response = fetch(host, path, expected_missing: true)
+        error(host, path, "page.soft_404", "unknown path returned #{response.status} (expected a missing-page response)") if (200..299).cover?(response.status)
+      end
+
+      def origin_for(host)
+        @config.origin_for(host)
+      end
+
       # @return [Rack::MockResponse]
-      def fetch(host, path, headers = {})
-        origin = URI.parse(@config.origin_for(host))
-        scheme = origin.scheme || "https"
-        # Rails HostAuthorization (config.hosts) reads the Host header, not SERVER_NAME: send both, with the port
-        # only when it is not the scheme's default.
-        authority = origin.port && origin.port != origin.default_port ? "#{host}:#{origin.port}" : host
-        env = Rack::MockRequest.env_for("#{scheme}://#{authority}#{path}",
-                                        { "REMOTE_ADDR" => "127.0.0.1", "HTTP_ACCEPT" => "text/html" }.merge(headers))
-        env["HTTP_HOST"] = authority
-        env["HTTPS"] = "on" if scheme == "https"
-        status, response_headers, body = @app.call(env)
-        response = Rack::MockResponse.new(status, response_headers, body)
-        if status == 403 && host_blocked?(host, response)
-          error(host, path, "host_authorization", "Rails HostAuthorization blocked Host: #{authority}; run it through " \
+      def fetch(host, path, headers = {}, expected_missing: false)
+        response = @fetcher.fetch(host, path, headers)
+        if response.status == 403 && host_blocked?(host, response)
+          error(host, path, "host_authorization", "Rails HostAuthorization blocked Host: #{URI.parse(origin_for(host)).authority}; run it through " \
                                                   "`bin/rails seo_geo:audit` (or set SEO_GEO_AUDIT=1 before boot, e.g. for " \
                                                   "bin/rails runner) so seo_geo allows its hosts outside production, or " \
                                                   "add #{host.inspect} to config.hosts")
         end
         response
       rescue StandardError => e
+        # Test/development apps can raise their configured 404 exceptions instead of rendering a response.
+        if expected_missing && defined?(ActionDispatch::ExceptionWrapper) &&
+           ActionDispatch::ExceptionWrapper.status_code_for_exception(e.class.name) == 404
+          return Rack::MockResponse.new(404, {}, [])
+        end
+
         error(host, path, "exception", "#{e.class}: #{e.message.lines.first.to_s.strip}")
         Rack::MockResponse.new(500, {}, [])
       end

data/lib/seo_geo/audit.rb

@@ -11,4 +11,7 @@ require_relative "audit/report"
 require_relative "audit/html_document"
 require_relative "audit/checks"
 require_relative "audit/static_scan"
+require_relative "audit/rack_fetcher"
+require_relative "audit/http_fetcher"
 require_relative "audit/runner"
+require_relative "audit/remote"

data/lib/seo_geo/controller.rb

@@ -33,6 +33,8 @@ module SeoGeo
     # @param canonical [String, false] path or URL to canonicalize; false omits the canonical link and hreflang
     # @param page [Symbol, Page] page copy (title/description/FAQ JSON-LD defaults)
     # @param type [String] og:type (default "website")
+    # @param published_time [Time, Date, String] ISO 8601 article publication time (type: "article" only)
+    # @param modified_time [Time, Date, String] ISO 8601 article modification time (type: "article" only)
     # @param markdown [Boolean] false hides the Markdown alternate link
     # @param json_ld [Hash, JsonLd, Array] extra JSON-LD nodes
     # @return [Meta::State]

data/lib/seo_geo/meta.rb

@@ -3,6 +3,8 @@
 require "uri"
 require "erb"
 require "json"
+require "time"
+require "date"
 require "meta-tags"
 
 module SeoGeo
@@ -21,7 +23,7 @@ module SeoGeo
   class Meta
     # Per-request values set by the controller ({SeoGeo::Controller#seo}) or views.
     State = Struct.new(:title, :description, :image, :noindex, :index, :canonical, :page, :type,
-                       :markdown, :json_ld, keyword_init: true) do
+                       :markdown, :json_ld, :published_time, :modified_time, keyword_init: true) do
       def initialize(**)
         super
         self.json_ld ||= []
@@ -168,6 +170,10 @@ module SeoGeo
       end
       tags[:og] = open_graph
       tags[:twitter] = twitter
+      if state.type.to_s == "article"
+        article = { published_time: article_time(state.published_time), modified_time: article_time(state.modified_time) }.compact
+        tags[:article] = article unless article.empty?
+      end
       tags.merge(config.verification.meta_tags.transform_keys(&:to_sym)).compact
     end
 
@@ -222,6 +228,17 @@ module SeoGeo
       links.empty? ? nil : links
     end
 
+    def article_time(value)
+      case value
+      when nil then nil
+      when Time, Date then value.iso8601
+      when String
+        value.match?(/\A\d{4}-\d{2}-\d{2}\z/) ? Date.iso8601(value).iso8601 : Time.iso8601(value).iso8601
+      else
+        raise ArgumentError, "article time must be a Time, Date or ISO 8601 string"
+      end
+    end
+
     def open_graph
       image = image_url
       {

data/lib/seo_geo/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SeoGeo
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

data/lib/tasks/seo_geo.rake

@@ -9,8 +9,19 @@ namespace :seo_geo do
   end
 
   desc "Audit robots.txt, sitemap.xml, llms.txt and every sitemap page in-process " \
-       "(FORMAT=json, HOSTS=a.com,b.com, MAX_PAGES=200); exits 1 when there are errors"
-  task audit: %i[audit_flag environment] do
+       "(URL=https://example.com or URLS=a,b for remote; FORMAT=json, HOSTS=a.com,b.com, MAX_PAGES=200); exits 1 when there are errors"
+  task :audit do
+    urls = (ENV["URLS"] || ENV["URL"]).to_s.split(",").map(&:strip).reject(&:empty?)
+    unless urls.empty?
+      require "seo_geo"
+      report = SeoGeo::Audit::Remote.run(urls, max_pages: Integer(ENV.fetch("MAX_PAGES", "200"), 10),
+                                       format: ENV.fetch("FORMAT", "text"))
+      exit(1) if report.errors?
+      next
+    end
+
+    Rake::Task["seo_geo:audit_flag"].invoke
+    Rake::Task["environment"].invoke
     report = SeoGeo::Audit::Runner.new(
       app: Rails.application,
       root: Rails.root,

data/llms.txt

@@ -2,9 +2,11 @@
 
 > Rails engine for SEO and GEO (Generative Engine Optimization: visibility in AI search and assistants), not a geolocation gem. One initializer and one layout line give meta tags with canonical folding, JSON-LD from one page-copy source, crawler fallback bodies for Inertia/SPA pages, purpose-based AI crawler policy (training / search / user), per-host robots.txt, sitemap.xml, llms.txt, Markdown negotiation, IndexNow, OG images and an audit task.
 
-- Install: `gem "seo_geo"` in the Gemfile, then `bin/rails g seo_geo:install https://www.example.com`. Version 0.1.0 is not yet on RubyGems; until it is, use `gem "seo_geo", github: "seunghan91/seo_geo"`.
+- Install: `gem "seo_geo"` in the Gemfile, then `bin/rails g seo_geo:install https://www.example.com`. Version 0.2.0 is on RubyGems.
 - Requirements: Ruby >= 3.2, Rails 7.2 / 8.0 / 8.1 (CI covers every cell). Runtime dependencies: railties, actionpack, meta-tags ~> 2.24, reverse_markdown ~> 3.0. Optional: ferrum (OG cards), ActiveJob (IndexNow job).
-- Public API: `SeoGeo.configure`, `SeoGeo.config`, `SeoGeo.sitemap.draw { add ... }`, `SeoGeo.page(:key)`, `SeoGeo.pages.register`; view helpers `seo_geo_head`, `seo_geo_html_lang`, `seo_geo_fallback_body(:key)`, `seo_geo_faq(:key)`; controller `seo(title:, description:, image:, noindex:, index:, canonical:, type:, json_ld:)`, `seo_geo_page(:key)`; `include SeoGeo::Markdownable` + `seo_geo_markdown only:, page:`; `SeoGeo::JsonLd`; `SeoGeo::IndexNow.submit(urls)`, model macro `seo_geo_indexnow on: :publish, url:`; `rake seo_geo:audit`; `rails g seo_geo:install`; `require "seo_geo/testing"` (Minitest) and `require "seo_geo/rspec"`; `SeoGeo::BotVerifier::Middleware`; JS `window.SeoGeo.mounted()`.
+- Public API: `SeoGeo.configure`, `SeoGeo.config`, `SeoGeo.sitemap.draw { add ... }`, `SeoGeo.page(:key)`, `SeoGeo.pages.register`; view helpers `seo_geo_head`, `seo_geo_html_lang`, `seo_geo_fallback_body(:key)`, `seo_geo_faq(:key)`; controller `seo(title:, description:, image:, noindex:, index:, canonical:, type:, json_ld:, published_time:, modified_time:)`, `seo_geo_page(:key)`; `include SeoGeo::Markdownable` + `seo_geo_markdown only:, page:`; `SeoGeo::JsonLd`; `SeoGeo::IndexNow.submit(urls)`, model macro `seo_geo_indexnow on: :publish, url:`; `rake seo_geo:audit`; `rails g seo_geo:install`; `require "seo_geo/testing"` (Minitest) and `require "seo_geo/rspec"`; `SeoGeo::BotVerifier::Middleware`; JS `window.SeoGeo.mounted()`.
+- Article times: `seo type: "article", published_time:, modified_time:` accepts Time, Date or ISO 8601 strings and emits ISO 8601 `article:published_time` / `article:modified_time`; omitted for other types. Use real stored dates.
+- Remote audit: `bundle exec rake seo_geo:audit URL=https://example.com` or `URLS=a,b` (FORMAT=json supported), no app boot; `SeoGeo::Audit::Remote.run(urls, io:)`. Serial Net::HTTP, 0.5s spacing, 10s timeouts, original redirect responses retained with at most five traced hops. Errors for header noindex and incomplete chains; warnings for >1 redirect hop and robots differences when matching config is available. Both modes check soft-404 and duplicate descriptions.
 - Routes served when mounted at "/": `GET /robots.txt`, `/sitemap.xml`, `/llms.txt`, `/llms-full.txt`, `/<indexnow-key>.txt`, `/og/:key.png`, and `.md` for every Markdownable action.
 - Invariants: undeclared hosts fail closed; one copy source per page; the User-Agent is never used for access decisions; no invented lastmod.
 - License: MIT.

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: seo_geo
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - seunghan Kim
 bindir: bin
 cert_chain: []
-date: 2026-10-14 00:00:00.000000000 Z
+date: 2026-10-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: railties
@@ -156,6 +156,9 @@ files:
 - lib/seo_geo/audit.rb
 - lib/seo_geo/audit/checks.rb
 - lib/seo_geo/audit/html_document.rb
+- lib/seo_geo/audit/http_fetcher.rb
+- lib/seo_geo/audit/rack_fetcher.rb
+- lib/seo_geo/audit/remote.rb
 - lib/seo_geo/audit/report.rb
 - lib/seo_geo/audit/runner.rb
 - lib/seo_geo/audit/static_scan.rb