indexmap 0.8.2 → 0.9.0

data/docs/dynamic-sitemaps.md

@@ -0,0 +1,302 @@
+# Database-backed sitemaps
+
+> This guide describes unreleased main. For RubyGems 0.8.2, use the
+> [tagged documentation](https://github.com/ethos-link/indexmap/tree/v0.8.2).
+> See [release status](../README.md#release-status) and [UPGRADE.md](../UPGRADE.md).
+
+Indexmap generates sitemap artifacts from app-owned sources. Generation can run
+in a job or rake task; HTTP requests serve published artifacts and never query
+source records. No model concern, queue backend, schedule, or content eligibility
+policy is installed by the gem.
+
+## Configure generation
+
+Run `bin/rails generate indexmap:install`, then edit the generated configure block:
+
+```ruby
+config.base_url = -> { ENV.fetch("PUBLIC_ORIGIN") }
+config.storage = Indexmap::Storage::Filesystem.new(
+  path: Rails.root.join("storage/sitemaps"), public_url: config.base_url
+)
+config.atomic = true
+config.max_urls = 45_000
+config.sections = -> { Sitemap.sections }
+```
+
+`Sitemap` is the application's manifest. It should return fresh sections whose
+entries are arrays or enumerables. Use native `find_each`/`in_batches`, selected
+columns, and SQL aggregates to select eligible records. Calculate canonical URLs
+with app-owned routing rules. Do not serialize a relation or URL inventory into
+a job, reuse an exhausted source across refreshes, or use refresh time as the
+content's `lastmod`. Time-dependent eligibility needs recurring refreshes even
+when no record is updated. A batch scan is not a point-in-time database snapshot.
+
+Generation stages XML on local temporary disk, validates the complete graph, then
+publishes. Entry sources are consumed once. Index mode splits a section before
+`max_urls` (default 50,000) or `max_bytes` (default 52,428,800) is exceeded. Limits
+must be positive integers within the protocol limits. Size includes escaped UTF-8
+XML and document overhead. Locations must be shorter than 2,048 characters.
+A single oversized entry fails; a single-file output
+that needs another partition fails with instructions to use index mode. Empty
+sections are omitted; an empty root fails without replacing the published root.
+The index itself must fit the protocol entry limit and configured byte limit.
+
+In legacy mode, the first partition keeps the section filename; later partitions
+append `-002.xml`, `-003.xml`, etc. Atomic mode chooses immutable child filenames
+instead; the configured root filename remains stable. Do not infer eligibility,
+content changes, or section identity from generated child filenames.
+
+For filesystem and Active Storage destinations, generation and validation retain
+at most a partition plus bounded parsing/sort buffers in memory, along with
+metadata proportional to the number of sitemap files. Duplicate checking uses
+10,000-URL sorted runs and a disk merge, including duplicates across partitions.
+Temporary disk use scales with the XML and URL inventory; provision enough local
+disk and monitor it. Validation and IndexNow planning use streaming reads and
+private temporary spools. Submission summaries retain per-batch outcomes and
+failures, so that metadata grows with request count. `Validator#each_entry` yields
+only after the entire inventory validates. `Validator#validate!` collects entries by default for Ruby
+callers; use `collect: false` when the list is not needed.
+
+The low-level `Writer#write`, `Parser#entries`, explicit `to_a` calls, memory
+storage, and app-owned arrays can still retain the whole inventory. Use
+`Writer#each_file` or the normal generation operation for incremental output.
+Custom adapters without `open`/metadata or `write_io` fall back to whole-file
+reads/writes; each file is still bounded by the configured limits. There is no
+full-document formatting pass during generation. Task reports use counts recorded
+while writing rather than downloading published files again. Serialization may
+differ from older Nokogiri-reformatted output while retaining XML meaning.
+
+## Atomic publication and storage
+
+Atomic mode is opt-in and requires `revision(root)` and
+`publish(source, files:, root:, expected:)` capabilities. An unsupported adapter
+fails before sources are evaluated. See the [complete storage contract](storage.md)
+before implementing a custom adapter. Existing six-method adapters keep legacy
+sequential publication unless explicitly extended. Named outputs inherit atomic
+mode and limits and can override them independently.
+
+Both built-in persistent adapters upload immutable children before activating the
+stable root. A root revision captured before generation must still match at
+activation. Otherwise `Indexmap::SupersededPublication` is raised and that build
+cannot overwrite a root published by another writer. Retry by rebuilding current
+eligible data. Queue concurrency reduces wasted work but does not replace this
+check, including for direct rake invocations.
+
+Filesystem publication copies into private destination staging without holding
+the per-root lock, including the previous root archive. Short locked sections
+establish a staging lease and activate complete files with same-filesystem
+renames; retirement timestamps are set at activation. Pruning skips live build
+leases and reclaims abandoned staging after the grace period. Exclude hidden
+`.indexmap-builds/` paths from HTTP serving and provision destination staging
+space. The destination must provide cross-process `flock` and atomic rename
+semantics; arbitrary NFS mounts or object-store filesystem gateways may not.
+Storage keys cannot escape through traversal or existing symlinks. Published
+immutable child keys cannot be overwritten through built-in write APIs.
+
+For Active Storage, keep the host-owned artifact model with `has_one_attached`
+and a database unique index on its filename column. Install Active Storage through
+Rails' installer and generate any host migration with Rails; inspect and remove
+existing duplicate rows before adding that index. Indexmap does not invent host
+migrations or modify attachment configuration. Example generator command:
+
+```sh
+bin/rails generate migration AddUniqueIndexToSitemapArtifactsFilename
+```
+
+Edit the generated migration to add the unique index after checking existing
+schema and data; do not add a duplicate index if one already exists. Atomic mode
+checks that an unconditional single-column unique index exists. Root blobs carry
+the generation identifier in native Active Storage metadata and are uploaded
+before a short host-record lock attaches them. Expected revisions are checked
+before uploads too; failed or losing builds immediately clean up the child
+records and blobs they created. A race can still waste an in-flight upload, but
+normal failure cleanup does not wait for pruning. Reading the active generation
+requires no object-store download under that lock. Generation and uploads run
+outside that database transaction. SQLite lock contention can raise a
+transient database error; preserve the host's explicit retry policy.
+
+The adapter archives the previous atomic sitemap index before activation. Archive ownership
+proves that a retained child's generation was published; uploaded orphans are
+excluded from shared HTTP delivery. Old children remain available for cached
+roots. Active Storage archive age follows the artifact record, not the older
+blob's creation date. The model's normal attachment lifecycle handles purging
+when expired artifact records are destroyed. Ensure its purge jobs have workers;
+use the host's existing unattached-blob cleanup for process crashes. Cleanup
+failures raise `Indexmap::PublicationCleanupError` with failed artifact/key
+diagnostics and the original publication exception as its cause. Preserve those
+private diagnostics: if object deletion fails after a blob row is destroyed,
+cleanup by object key may be necessary. Host callback errors after a committed
+activation do not delete that published generation.
+
+Keep the destination shared by workers and every serving instance. A container's
+`public/` directory is not automatically shared or persistent. In legacy mode,
+validation failure preserves old artifacts, but interrupted sequential writes can
+still leave mixed generations. Atomic mode solves that collection-level problem.
+
+## Jobs and schedules
+
+Generate the job in the host application, then edit it:
+
+```sh
+bin/rails generate job SitemapRefresh
+```
+
+```ruby
+require "indexmap"
+
+class SitemapRefreshJob < ApplicationJob
+  retry_on Indexmap::SupersededPublication, wait: :polynomially_longer, attempts: 3
+
+  def perform(_version = nil)
+    Indexmap::HostConfiguration.apply
+    Indexmap::TaskRunner.new.create
+  end
+end
+```
+
+Load the host configuration file first if it lives outside an initializer. The
+explicit require above makes the exception class available when a job is
+eager-loaded; it does not evaluate section sources. The host's `ApplicationJob` context,
+authorization, and transaction hooks remain authoritative. Pass scalar output or
+release identifiers and enqueue after source transactions commit and database
+preparation completes. A separate queue database is not transactionally coupled
+to application data.
+
+The same body works with GoodJob and Solid Queue. Keep the adapter-specific
+concurrency declarations and recurring schedules in the host; see the
+[GoodJob and Solid Queue examples](../README.md#background-generation). Use a
+shared execution key/label across releases for the same destination. Solid Queue
+uses a shared concurrency group too if multiple job classes write that destination.
+Prefer blocking/rescheduling to discarding a newer refresh. Choose a concurrency
+duration above measured runtime and remember it can expire. Direct `perform_now`
+and rake calls bypass queue coordination but still use atomic activation checks.
+
+Use one enqueue owner per trigger. Avoid duplicate startup/release scheduling,
+per-country fan-out, and continuations until actual runtime justifies them. Retry
+identified transient storage/database errors deliberately; do not repeatedly retry
+invalid configuration or content. `after_create` callbacks run after publication
+and must be safe to repeat. A callback failure does not roll publication back.
+
+`TaskRunner#create` runs default-output callbacks and key-file handling.
+`Indexmap.create(:name)` generates a named output without those callbacks. Keep
+that distinction when designing independent refresh jobs. Generation does not
+submit to IndexNow; keep sending as a separate operation so refresh retries do
+not resend successful batches.
+
+## Serve published XML
+
+Require the controller and explicitly mount the root and generated child pattern.
+This example assumes root-level storage keys and `index_filename = "sitemap.xml"`:
+
+```ruby
+require "indexmap/sitemaps_controller"
+Indexmap::HostConfiguration.apply
+
+Rails.application.routes.draw do
+  get "/sitemap.xml", to: Indexmap::SitemapsController.action(:show),
+    defaults: {filename: "sitemap.xml"}, format: false
+  get "/:filename", to: Indexmap::SitemapsController.action(:show),
+    constraints: {filename: /sitemap-[a-f0-9]{16}-[a-f0-9]{32}-[1-9]\d*-[1-9]\d*\.xml/},
+    format: false
+end
+```
+
+Mount named roots explicitly too. For nested keys, set the full root key in the
+route defaults and a fixed `prefix` default for child routes. For example a
+`/sitemaps/:filename` child route can use `defaults: {prefix: "sitemaps"}`. Only
+path parameters are used, so query strings cannot supply a storage prefix. Dots
+must be accepted by the filename constraint; disabling format parsing alone does
+not make Rails dynamic segments accept dots. Do not mount a broad catchall route.
+
+The controller allows configured roots and existing children from current or
+archived atomic publications. It never resolves section providers. Other files,
+archive indexes, and unpublished orphan children are excluded. Delivery requires
+storage `metadata` and `open` capabilities, implemented by both persistent
+adapters. The endpoint uses `application/xml`, streams bodies, and supports HEAD,
+ETag, Last-Modified, and 304. Roots cache for ten minutes with one hour of stale
+revalidation; immutable children cache for a year. Generation authorization uses
+the host's Rails cache keyed by the current root's metadata. Authentication,
+visitor tracking, and browser gates from host controllers are not inherited.
+
+When switching from legacy child names, keep the old allowlisted child routes
+or static files through the cached-root/rollback grace period. Atomic mode does
+not rename or prune those legacy files automatically, and the new shared
+controller does not authorize arbitrary legacy children. Add the generated child
+route first, enable atomic publication, and retire legacy serving only after the
+transition period. Then remove the old child artifacts deliberately.
+
+Remove static files that would shadow these routes, or deliberately keep static
+serving and configure its root/child cache headers appropriately. Robots is a
+separate opt-in endpoint; see [dynamic robots](../README.md#dynamic-robots).
+Public origins must come from configuration, not request headers. Storage public
+URLs must be the URLs that the app actually serves, not private object-store URLs.
+
+## Retention and operations
+
+```sh
+bin/rails indexmap:sitemap:create
+bin/rails indexmap:sitemap:validate
+RETENTION_HOURS=48 bin/rails indexmap:sitemap:prune
+```
+
+Schedule pruning separately from generation. The default grace period is 48 hours;
+values below one hour are rejected. Choose a period longer than the oldest root
+that your caches can still serve, including stale revalidation and deployment
+rollbacks. Retention is an operational policy, not a promise about arbitrary
+crawler caches. Pruning is locked against activation, preserves the current
+publication, removes expired archives and unreferenced owned children, and leaves
+other outputs and unrelated files alone. Abandoned destination staging and
+crash-left orphans age out too. Live filesystem staging remains protected by its
+build lock. Active Storage activation rechecks child existence, so pruning an
+old in-flight build forces a rebuild instead of activating an incomplete root.
+Active Storage pruning destroys owned artifact records and delegates blob cleanup
+to normal attachment jobs, avoiding object-store deletes under the root lock.
+
+Do not use an old app callback that deletes everything outside configured section
+filenames: atomic child names deliberately differ. Replace it with the gem's
+prune task when adopting atomic mode. The format task refuses to run when an
+atomic output is configured. Regenerate instead of mutating immutable XML.
+
+For IndexNow selection, exact attempt exports, timeouts, and bounded retries, see
+[search-engine submission](search-engine-ping.md#ping-indexnow). A missing timestamp
+is not evidence of a content change, and removal from a sitemap is not evidence
+of deletion. Durable history and automatic inventory diffs are not implemented.
+
+## Host adoption
+
+Release and consume the gem before removing host implementations. Local path
+overrides are for verification only and must not be committed to applications.
+
+1. Keep eligibility scopes, canonical URLs, content dates, and scheduling in the
+   host. Static manifests can stay database-free; database sources should return
+   fresh lazy enumerables.
+2. Choose storage shared by refresh workers and serving instances. Verify the
+   artifact filename index when using atomic Active Storage.
+3. Add explicit root and generated-child routes, or configure static/CDN delivery.
+   Preserve legacy child delivery through the cached-root transition.
+4. Replace filename-based cleanup callbacks with output-scoped pruning. Preserve
+   the host's robots exclusions and production/preview indexing policy.
+5. Verify the lifecycle in development, then staging, before enabling public
+   indexing and search-engine submission. See the [deployment guide](deployment.md).
+
+Validate actual public root/child URLs, headers, key verification files, workers,
+and queue failure reporting in each host. Gem integration tests exercise real
+GoodJob, Solid Queue, and Active Storage against PostgreSQL; they do not prove a
+particular application's deployment wiring.
+
+## Integration checks from a source checkout
+
+The integration harness creates a temporary Rails app and runs framework/gem
+installers and generators. It drops and recreates the supplied `indexmap_test`
+database. Never point it at an application database.
+
+```sh
+BUNDLE_GEMFILE=gemfiles/integration.gemfile bundle install
+INDEXMAP_DATABASE_URL=postgresql://postgres@localhost/indexmap_test \
+  BUNDLE_GEMFILE=gemfiles/integration.gemfile bundle exec ruby script/integration
+```
+
+The harness checks the real GoodJob adapter, a running Solid Queue worker,
+validation failure visibility, shared filesystem output, Active Storage activation,
+and archive retention. CI provides a disposable PostgreSQL service. No external
+search-engine requests are made.

data/docs/search-engine-ping.md

@@ -0,0 +1,368 @@
+# Search Engine Ping
+
+> This guide describes unreleased main. For RubyGems 0.8.2, use the
+> [tagged documentation](https://github.com/ethos-link/indexmap/tree/v0.8.2).
+> See [release status](../README.md#release-status) and [UPGRADE.md](../UPGRADE.md).
+
+`indexmap` can submit generated sitemap data to Google Search Console and
+IndexNow. These integrations are optional: sitemap generation, formatting, and
+validation work without any search-engine credentials.
+
+Available rake tasks:
+
+```bash
+bin/rails indexmap:sitemap:validate
+bin/rails indexmap:google:ping
+bin/rails indexmap:index_now:ping
+bin/rails indexmap:index_now:write_key
+bin/rails indexmap:ping
+```
+
+`indexmap:ping` runs both IndexNow and Google submission. Run
+`indexmap:sitemap:create` before pinging so the configured storage contains the
+current sitemap files.
+
+The configured `index_filename` is the authoritative sitemap entrypoint.
+Google submits that sitemap URL once. IndexNow recursively reads its referenced
+child sitemaps and submits their page URLs. Other sitemap-shaped files left in
+storage are ignored unless the authoritative sitemap references them.
+
+## Google Search Console
+
+Google submission uses the Search Console Sitemaps API. It submits sitemap URLs
+such as `https://example.com/sitemap.xml` to a verified Search Console
+property. It does not submit arbitrary page URLs to Google for immediate
+indexing.
+
+Current Google API contract:
+
+- API: Search Console Sitemaps API
+- Method: `PUT https://www.googleapis.com/webmasters/v3/sites/{siteUrl}/sitemaps/{feedpath}`
+- OAuth scope: `https://www.googleapis.com/auth/webmasters`
+- Request body: empty
+- Success response body: empty
+
+The Google Indexing API is a separate page-level API at
+`POST https://indexing.googleapis.com/v3/urlNotifications:publish` with the
+`https://www.googleapis.com/auth/indexing` scope. Google documents that API for
+JobPosting pages and livestreaming events with `BroadcastEvent` structured data,
+not for general sitemap submission. `indexmap` therefore uses the Search Console
+Sitemaps API for Google.
+
+### Create The Google Credentials
+
+1. Open or create a Google Cloud project for the site.
+2. Enable the Google Search Console API for that project.
+3. Create a service account for sitemap submission.
+4. Create a JSON key for that service account.
+5. Store the downloaded JSON as a secret in the host app. Do not commit it.
+6. Copy the JSON key's `client_email`.
+7. Open the Search Console property for the site.
+8. Go to `Settings` > `Users and permissions`.
+9. Add the service account `client_email` as a user with permission to submit
+   sitemaps. `Full` user access is sufficient for sitemap submission; `Owner`
+   also works but grants broader access.
+
+The Search Console property must match what `indexmap` submits. Domain
+properties use `sc-domain:example.com`; URL-prefix properties use the exact URL
+prefix shown in Search Console, such as `https://www.example.com/`.
+
+### Configure Indexmap
+
+`config.google.credentials` must return the JSON key contents, not a path to a
+file.
+
+Rails credentials example:
+
+```yaml
+google:
+  sitemap: |
+    {
+      "type": "service_account",
+      "project_id": "example-project",
+      "private_key_id": "...",
+      "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
+      "client_email": "indexmap-sitemap@example-project.iam.gserviceaccount.com",
+      "client_id": "...",
+      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
+      "token_uri": "https://oauth2.googleapis.com/token",
+      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
+      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/indexmap-sitemap%40example-project.iam.gserviceaccount.com"
+    }
+```
+
+Initializer example:
+
+```ruby
+Indexmap.configure do |config|
+  config.google.credentials = -> { Rails.application.credentials.dig(:google, :sitemap) }
+  config.google.property = -> { "sc-domain:example.com" }
+end
+```
+
+Environment variable example:
+
+```ruby
+Indexmap.configure do |config|
+  config.google.credentials = -> { ENV["GOOGLE_SITEMAP_JSON"] }
+  config.google.property = -> { ENV.fetch("GOOGLE_SEARCH_CONSOLE_PROPERTY", "sc-domain:example.com") }
+end
+```
+
+If `config.google.property` is not set, `indexmap` defaults to
+`sc-domain:<host>` from `config.base_url`.
+
+Run:
+
+```bash
+bin/rails indexmap:sitemap:create
+bin/rails indexmap:sitemap:validate
+bin/rails indexmap:google:ping
+```
+
+A successful ping prints how many sitemap files and URLs were submitted. If the
+task says Search Console does not have access, compare the configured property
+identifier and the service account `client_email` against the Search Console
+property's `Users and permissions` page.
+
+## IndexNow
+
+IndexNow submission sends changed URL lists to the IndexNow endpoint. It is a
+separate protocol from Google Search Console submission.
+
+IndexNow requires a key that proves control of the submitted host. `indexmap`
+uses a strict key format: 32 lowercase hexadecimal characters. The verification
+file must be named `<key>.txt` unless `config.index_now.key_filename` overrides
+the filename, and its contents must be exactly the key.
+
+### File-Based Key Provisioning
+
+The simplest flow is to let `indexmap` generate or reuse the key file:
+
+```bash
+bin/rails indexmap:index_now:write_key
+```
+
+That task:
+
+- reuses an existing valid `<key>.txt` file from the configured storage
+- otherwise generates a new 32-character hexadecimal key
+- writes `<key>.txt` with the key as the file contents
+- makes that key available to `indexmap:index_now:ping`
+
+After deployment, confirm the key file is publicly reachable:
+
+```bash
+curl -fsS https://example.com/<key>.txt
+```
+
+The response body should be the key only, with no extra text.
+
+Key discovery only searches `config.storage`. For example, when sitemap storage
+points at `storage/sitemaps`, a verification file in Rails' `public/` directory
+will not be discovered. Use the configuration below for that arrangement.
+
+### Configured Key
+
+If the host app already stores an IndexNow key in credentials or environment
+variables, configure it directly:
+
+```ruby
+Indexmap.configure do |config|
+  config.index_now.key = -> { ENV["INDEXNOW_KEY"] }
+end
+```
+
+When `config.index_now.key` is set, `indexmap:sitemap:create` also ensures the
+matching `<key>.txt` verification file exists in the configured storage. It
+leaves an existing valid key file unchanged.
+
+If the verification file uses a non-standard name, configure it explicitly:
+
+```ruby
+Indexmap.configure do |config|
+  config.index_now.key = -> { ENV["INDEXNOW_KEY"] }
+  config.index_now.key_filename = -> { ENV.fetch("INDEXNOW_KEY") + ".txt" }
+end
+```
+
+If another part of the host app owns the verification file, disable automatic
+key-file writes:
+
+```ruby
+Indexmap.configure do |config|
+  config.index_now.key = -> { ENV["INDEXNOW_KEY"] }
+  config.index_now.write_key_file = false
+end
+```
+
+### Rails Public Key File With Separate Sitemap Storage
+
+If Rails serves an existing verification file from `public/` while sitemaps
+live in `storage/sitemaps` or another backend, configure the key explicitly and
+keep the host app responsible for serving the file:
+
+```ruby
+Indexmap.configure do |config|
+  config.base_url = -> { "https://www.example.com" }
+  config.storage = lambda {
+    Indexmap::Storage::Filesystem.new(
+      path: Rails.root.join("storage/sitemaps"),
+      public_url: config.base_url
+    )
+  }
+  config.index_now.key = -> {
+    Rails.public_path.join("0123456789abcdef0123456789abcdef.txt").read.strip
+  }
+  config.index_now.write_key_file = false
+end
+```
+
+Replace the example filename with your existing `<key>.txt` filename. For a new
+key, generate one with `ruby -rsecurerandom -e 'print SecureRandom.hex(16)'`, then
+create `public/<key>.txt` containing exactly that key, without a trailing newline.
+Deploy the file with the application.
+
+`write_key_file = false` prevents sitemap generation from writing a second key
+file into sitemap storage. It does not change where the pinger searches for a
+key, so `config.index_now.key` must also be set.
+
+The pinger builds `keyLocation` using
+`config.storage.public_url(config.index_now.key_filename)`. In this example,
+that URL is `https://www.example.com/<key>.txt`. Ensure the storage adapter
+returns the actual public verification URL; adapters with a different public
+origin or path need a matching publicly served file there.
+
+The host app must also serve the sitemap files at their configured public URLs.
+A filesystem adapter writes files but does not create Rails routes for them.
+
+Before submitting URLs, verify the deployed file and run a dry run in the
+production environment:
+
+```bash
+curl -fsS https://www.example.com/<key>.txt
+RAILS_ENV=production bin/rails indexmap:sitemap:create
+RAILS_ENV=production bin/rails indexmap:sitemap:validate
+RAILS_ENV=production INDEXNOW_DRY_RUN=1 bin/rails indexmap:index_now:ping
+```
+
+The curl response must contain exactly the configured key. A dry run checks
+local key loading and sitemap parsing; it does not verify that IndexNow can
+fetch the public file. After those checks, run
+`RAILS_ENV=production bin/rails indexmap:index_now:ping` to submit the URLs.
+
+### Ping IndexNow
+
+#### Inspect selection and actual attempts
+
+Preview the exact selection without submitting or requiring a key:
+
+```bash
+SINCE=2026-04-18T10:30:00Z bin/rails indexmap:index_now:plan
+```
+
+The task writes one JSON object per URL, including excluded URLs. Each row has
+`loc`, `lastmod`, `source_sitemap`, `index_filename`, the resolved UTC `cutoff`,
+`selected`, `reason`, the selected `batch` number, and `missing_key`. Reasons are
+`all`, `recent`, `missing_lastmod`, and `before_cutoff`. An empty root discovery
+produces no rows. Invalid reachable sitemaps raise before selection or sending.
+Planning reads configured storage only; it makes no network requests or writes.
+
+To record which URLs a real submission attempts, use:
+
+```bash
+INDEXNOW_REPORT=1 bin/rails indexmap:index_now:ping > /private/operator-path/indexnow-attempts.jsonl
+```
+
+Use an existing private destination outside `public/` and sitemap storage.
+The report replaces the task's human-readable output with JSON Lines:
+`attempt_started` contains the batch number and exact `urls`; `attempt_finished`
+adds its status and HTTP receipt; `summary` retains the normal result totals.
+HTTP 202 has receipt `validation_pending`; other successful HTTP responses have
+receipt `received`. Neither proves crawling or indexing. Dry runs emit only a
+summary because they make no attempts.
+
+Timeouts and connection failures produce a failed batch with reason
+`unknown_delivery`. Earlier successful batches remain in the summary. There are no
+automatic retries by default. Requests use a five-second connection timeout and
+a 20-second request timeout. API keys are omitted from reports.
+
+Enable bounded retries explicitly in the existing configure block:
+
+```ruby
+config.index_now.max_attempts = 3
+config.index_now.max_wait = 30
+```
+
+Attempts must be an integer from 1 through 5; `max_wait` is a finite number of
+seconds from 0 through 60. The default is one attempt and a 30-second wait budget.
+Only the affected batch retries for transport uncertainty or HTTP 429, 500, 502,
+503, and 504. Backoff includes jitter. A valid `Retry-After` takes precedence;
+when it exceeds `max_wait`, the batch remains failed instead of retrying early.
+Permanent errors and HTTP 202 do not retry. Both report events include an
+`attempt` number, so an initially uncertain attempt remains visible even when a
+later attempt succeeds. Summary counts describe final batch outcomes.
+
+Do not stack whole-job retries around a partially successful submission: that can
+resend earlier successful batches. Retries do not survive process restarts and
+may duplicate a remotely received request after a timeout. Generation and
+submission remain separate. Preview and sending validate and spool the complete
+reachable inventory before yielding selection or issuing requests; selected URLs
+are then read in bounded batches instead of retaining an inventory-sized array.
+
+Exports are not a durable ledger. A preview and a later submission can differ
+when files or configuration change. An attempt-started record proves local
+intent, not remote receipt; a crash or timeout can leave delivery uncertain.
+Retain reports yourself if historical visibility is required.
+
+Ruby callers can use `pinger.plan.each` for selection rows and
+`pinger.ping { |event| ... }` for attempt events without retaining another URL
+inventory. Exceptions raised by the reporting block propagate to the caller.
+
+#### Submit selected URLs
+
+Run:
+
+```bash
+bin/rails indexmap:sitemap:create
+bin/rails indexmap:sitemap:validate
+bin/rails indexmap:index_now:ping
+```
+
+For a dry run:
+
+```bash
+INDEXNOW_DRY_RUN=1 bin/rails indexmap:index_now:ping
+```
+
+By default, `indexmap:index_now:ping` submits all current sitemap URLs. To limit
+the submission to recently changed sitemap entries, set one of:
+
+```bash
+SINCE=2026-04-18T10:30:00Z bin/rails indexmap:index_now:ping
+INDEXNOW_RECENT_HOURS=24 bin/rails indexmap:index_now:ping
+```
+
+`SINCE` takes precedence; invalid values raise rather than falling back. Date-only
+`lastmod` values are compared to the cutoff's UTC calendar date, including that
+whole day. Missing dates are always included, so this filter is not a precise
+content-change detector. Batch sizes default to 500 and must be integers from
+1 through 10,000, configured with `config.index_now.max_urls_per_request` or
+`INDEXNOW_MAX_URLS_PER_REQUEST`.
+
+If the task says the key is missing, either configure `config.index_now.key` or
+run `indexmap:index_now:write_key` and deploy the generated key file. If a key
+file already exists in `public/`, check whether `config.storage` points at a
+different directory; use the separate-storage example above in that case. If the
+remote endpoint returns `403`, verify that the public key file URL returns the
+same key that `indexmap` is submitting.
+
+## References
+
+- [Google Search Console Sitemaps API](https://developers.google.com/webmaster-tools/v1/sitemaps/submit)
+- [Google Search Console API reference](https://developers.google.com/webmaster-tools/v1/api_reference_index)
+- [Google Indexing API urlNotifications reference](https://developers.google.com/search/apis/indexing-api/v3/reference/indexing/rest/v3/urlNotifications/)
+- [Google Indexing API usage limits](https://developers.google.com/search/apis/indexing-api/v3/using-api)
+- [Google Cloud service account keys](https://cloud.google.com/iam/docs/keys-create-delete)
+- [Search Console users and permissions](https://support.google.com/webmasters/answer/7687615)
+- [IndexNow protocol documentation](https://www.indexnow.org/documentation)