indexmap 0.8.2 → 0.9.0

data/UPGRADE.md

@@ -0,0 +1,361 @@
+# Upgrade Guide
+
+Use this guide when upgrading an application that already generates, validates,
+or pings sitemaps with `indexmap`.
+
+## Unreleased
+
+### Host loader and rendered artifacts
+
+The install generator now defines `Indexmap::HostConfiguration.apply` instead of
+`IndexmapConfiguration.apply`. When adopting the new template, move the existing
+loader into `module Indexmap; module HostConfiguration; ...; end; end` and update
+its bottom-of-file apply call, route mounts, jobs, scripts, and tests. Keep the
+existing configure block, mutex, and idempotent `@applied` guard. Inside the new
+namespace, qualify host constants that overlap gem names (for example `::Entry`
+for a host model, versus `Indexmap::Entry` for a sitemap value). Use:
+
+```ruby
+Indexmap::HostConfiguration.apply if defined?(Indexmap::Configuration)
+```
+
+An existing host-owned loader can continue running while you migrate: the gem
+never calls it by name. Do not mix old and new loader modules, as each has its own
+application guard. Apply the host configuration before creating an operation.
+
+The rendered file value is now `Indexmap::Artifact`, replacing
+`Indexmap::Storage::File`. Its `filename`, `body`, `content_type`, and `basename`
+interface is unchanged. Update custom adapters, type checks, and direct
+constructor calls; replace internal `require "indexmap/storage/file"` with
+`require "indexmap"`. The old constant and require path are removed. `::File`
+always means Ruby's filesystem class.
+
+### Storage resolution and publication lifecycle
+
+`Creator`, `TaskRunner`, `Validator`, and pinger instances now resolve a storage
+callable lazily once and retain that adapter. Nested generation, validation,
+IndexNow key handling, and submission share it. Create a fresh operation instance
+after changing storage configuration or tenant/context; do not reuse a runner or
+pinger across different destinations. `Indexmap.create` already creates a new
+creator each time. The optional `storage:` constructor keyword selects an explicit
+adapter. Direct reads of `config.storage` still resolve the callable each time.
+
+Filesystem publication now copies validated files and the previous root into
+private `.indexmap-builds/` staging on the destination filesystem, outside the
+per-root lock. Short locked sections create the staging lease and activate files
+with same-filesystem renames. Keep that private directory writable and exclude
+hidden paths from static HTTP serving. Allow disk space for source staging,
+destination staging, and retained generations. Do not remove active staging
+directories manually. Pruning reclaims abandoned staging after its grace period
+and skips directories whose build lock is held; returned deleted paths can now
+include these private directories. Normal failures remove staging immediately.
+
+Filesystem listing now scopes traversal by the literal prefix's directory and
+leading filename fragment. It excludes symlinks and hidden paths, including
+private staging, and requires normalized relative prefixes (no traversal,
+absolute paths, or redundant separators). Replace any reliance on symlink entries with actual keys under the storage root. A listing without a
+prefix still intentionally scans the complete visible tree.
+
+Atomic Active Storage checks the expected revision before uploading and again
+before each upload. A failed or superseded build destroys its newly created child
+records and purges its blobs immediately, including a partial upload whose blob
+was saved before the service failed. Uploads and cleanup stay outside the root
+row lock. A committed root is preserved even if a host `after_commit` callback
+raises. Activation also checks that children still exist, so pruning an unusually
+old in-flight build aborts it rather than publishing missing children.
+
+Concurrent builds can still perform some speculative uploads before one wins;
+optimistic publication cannot avoid work already in flight. Process death and
+storage outages can also prevent immediate cleanup. Keep scheduled pruning,
+unattached-blob cleanup, and working attachment purge jobs. A
+`Indexmap::PublicationCleanupError` reports cleanup failures through `failures`
+(each contains `artifact` and `error`), with the original publication exception as
+its cause. Retain those private diagnostics; a failed service deletion can require
+cleanup by object key even when Active Storage already removed the blob row.
+
+### Maintainer release workflow
+
+`release:prepare[...]` now runs from clean, updated `main` matching
+`origin/main`. It updates the version and changelog, creates a Conventional
+Commit with hooks enabled, and pushes `main`. Replace scripts that prepare on a
+topic branch or separately commit those files. The branch hook permits only a
+release-task commit containing the bumped version and matching changelog; other
+changes still require a topic branch. Ensure remote branch protections permit
+the maintainer to push this release commit. Then run `bundle exec rake release:tag`
+to create and push the tag. If pushing preparation fails, push the existing
+release commit before tagging rather than preparing the same version again.
+See the [release instructions](README.md#release).
+
+### Task loading, parsing, and documentation
+
+For an existing lazy installation, edit the generated Rakefile loader to check
+Rake's collected tasks as well as `ARGV` (Rails removes the command from `ARGV`
+before loading the Rakefile):
+
+```ruby
+indexmap_tasks_requested = Rake.application.options.show_tasks ||
+  ARGV.any? { |argument| ["-T", "--tasks"].include?(argument) }
+require "indexmap/tasks" if indexmap_tasks_requested ||
+  (Rake.application.top_level_tasks + ARGV).any? { |argument| argument.start_with?("indexmap:") }
+```
+
+Keep this block before `require_relative "config/application"`. The installer
+preserves existing task loaders, so rerunning it does not replace a stale block.
+Verify `bin/rails indexmap:sitemap:create`, `bin/rails --tasks`, and
+`bundle exec rake indexmap:sitemap:validate`. The generated initializer now checks
+`defined?(Indexmap::Configuration)` before applying settings and constructs the
+filesystem adapter once inside `apply`. Apply those edits to existing initializers
+if needed, together with the host-loader rename above.
+
+`Parser` remote children and redirects must now keep the explicit source URL's
+scheme, host, and port; userinfo is rejected. Cross-origin references raise
+`Indexmap::ValidationError` before their target is requested. To inspect a deployed
+sitemap through a local origin, use `rebase_remote_children: true` deliberately;
+this rebases child references, not redirects. Pass only trusted remote root URLs:
+the parser is not an SSRF filter for arbitrary user-supplied roots or DNS results.
+Remote responses and local files are capped at 52,428,800 bytes per document, and
+DTDs are rejected. Streaming XML parsing can raise `Nokogiri::XML::SyntaxError`
+for malformed XML that the older DOM parser recovered silently. Fix malformed or
+oversized/DTD-bearing documents and cross-origin references before upgrading.
+Storage-root parsing resolves children through storage and does not fetch remote
+child URLs.
+
+Use `Parser#each_entry` for incremental enumeration without the `entries` cache;
+`entries` and `paths` still collect results. Google URL counting now streams XML
+and uses temporary disk for exact deduplication. Provision temporary space for
+that operation as well as generation/validation.
+
+Create stages, validates, and publishes; it does not pretty-print. Only
+`TaskRunner#create` and its rake task run callbacks and IndexNow key handling.
+The optional format task remains for legacy output. See the
+[deployment guide](docs/deployment.md) for static/dynamic sources and delivery,
+robots policy, and development/production checks. Main documentation describes
+unreleased features; use tagged documentation while consuming 0.8.2.
+
+### Generation, publication, and delivery
+
+Generation now stages XML on temporary disk and partitions index sections by
+`max_urls` and `max_bytes`. Defaults are the sitemap protocol limits. The first
+legacy partition keeps its configured name; later files append `-002.xml`, etc.
+Single-file mode still writes one file and fails if splitting would be required.
+Locations must be shorter than 2,048 characters; repeated `loc`/`lastmod` fields
+are rejected. Provision temporary disk for XML, validation spools, and duplicate
+sorting. The full-document Nokogiri formatting pass is removed; XML meaning is preserved but
+serialized bytes/checksums can change. Generation reports no longer download
+published artifacts to count URLs.
+
+Set `config.atomic = true` to opt into immutable generation children and guarded
+root activation. Filesystem and Active Storage implement the required revision
+and publish capabilities. Custom six-method adapters retain legacy behavior;
+unsupported atomic mode fails explicitly. Generated child keys use a reserved
+`sitemap-<owner>-<generation>-<section>-<part>.xml` namespace and cannot be
+rewritten by built-in adapters. Update fixed filename route constraints, remove
+static files that shadow new routes, and replace app cleanup callbacks that
+recompute configured filenames. They would delete generated children.
+
+Atomic Active Storage requires an unconditional unique filename index. Inspect
+existing data/schema and use a generated host migration if needed. The host
+artifact model remains app-owned. Retention uses archive record timestamps and
+the model's normal attachment cleanup, which requires working purge-job workers.
+Filesystem atomic mode requires shared lock/rename semantics; a worker-local
+container directory is insufficient for multiple serving instances.
+
+`Indexmap::SupersededPublication` means another build activated the root after
+this build began. Retry by rebuilding; do not fall back to sequential writes.
+Use native bounded Active Job retries in host jobs for this transient condition.
+The same entrypoints support GoodJob and Solid Queue. The opt-in sitemap
+controller serves configured roots and current/retained atomic children using
+storage metadata and streaming reads. See [dynamic sitemap usage](docs/dynamic-sitemaps.md)
+for routes, cache policy, adapter capabilities, and adoption steps.
+
+Schedule `indexmap:sitemap:prune` separately. `RETENTION_HOURS` defaults to 48 and
+must be at least 1; choose a grace period covering cached roots and rollbacks.
+The format task refuses to mutate atomic publications; regenerate instead.
+
+IndexNow selection now spools validated entries before sending, without retaining
+all selected URLs. Optional bounded retries use `config.index_now.max_attempts`
+(1..5, default 1) and `max_wait` (0..60 seconds, default 30). Attempt exports now
+include attempt numbers. Long `Retry-After` values defer retries rather than
+being shortened; this is not durable scheduling or exactly-once delivery.
+
+Section entries are now consumed once per generation, including calculation of
+the index's `lastmod`. Existing array sources keep the same XML meaning. Apps may
+return fresh lazy enumerables from `config.sections`; do not reuse exhausted
+one-shot sources between refreshes. Streaming generation and opt-in atomic
+publication are described above.
+
+Sitemap validation now rejects files larger than 52,428,800 uncompressed bytes,
+URL sets and indexes with more than 50,000 entries, missing or incorrect sitemap
+namespaces, DTDs, missing URL locations, cyclic indexes, unsafe child paths, and
+child URLs outside the configured scheme, host, or port. Relative local child
+references remain supported. Fix invalid existing files before upgrading.
+Generation still validates before publication; sequential storage writes do not
+provide atomic publication of the entire sitemap set.
+
+IndexNow now validates the reachable sitemap before planning or sending. Duplicate
+or otherwise invalid entries fail validation instead of being silently filtered.
+Index sections partition automatically; use index format when a single-file
+output would exceed its limits.
+
+Date-only `lastmod` values now participate in IndexNow cutoff selection. Dates on
+or after the cutoff's UTC calendar day are included, even when the cutoff has a
+later time on that day. Missing dates remain included. Invalid `SINCE` still
+raises and does not fall back to `INDEXNOW_RECENT_HOURS`. Batch sizes must be
+integers from 1 through 10,000; malformed configuration is no longer truncated.
+
+The new `indexmap:index_now:plan` task exports selection as JSON Lines without a
+key or network submission. Existing dry runs still require a key. Set
+`INDEXNOW_REPORT=1` on the ping task for JSON Lines containing attempt-started,
+attempt-finished, and summary records. These are non-durable exports: save them
+to private operator storage if needed, never public sitemap storage. HTTP 202
+is reported as `validation_pending` in the attempt receipt. The summary retains
+its existing `submitted` status for successful HTTP responses; it does not prove
+indexing. Transport failures now appear in partial/failed results with reason
+`unknown_delivery`, preserving earlier successful batches. Retries are opt-in as
+described above; there is no durable delivery history.
+
+Robots delivery is opt-in. Require `indexmap/robots_controller`, apply the host
+configuration, and mount the controller explicitly as documented in the README.
+Remove `public/robots.txt` when mounting it; static files can shadow Rails routes.
+Crawling defaults to disabled. Configure an explicit predicate that also handles
+staging running in production mode. The endpoint does not change host routes,
+queue adapters, refresh jobs, or sitemap storage.
+
+Search-engine ping tasks now treat `config.index_filename` as the authoritative
+sitemap entrypoint. Google submits that sitemap once, and IndexNow recursively
+reads the child sitemaps it references. Unreferenced `sitemap*.xml` files in the
+configured storage are no longer submitted or included in IndexNow URL batches.
+
+Before upgrading, ensure every current sitemap is reachable from the configured
+root sitemap. If an application intentionally maintains an independent named
+output, submit that output separately instead of relying on storage discovery.
+
+## Upgrading To 0.6.x
+
+Version 0.6.x routes every sitemap operation through the configured storage
+backend. Generation, formatting, validation, parsing, Google submission,
+IndexNow submission, and IndexNow verification files should all read and write
+through the same storage object.
+
+### 1. Configure Storage Explicitly
+
+Set `config.storage` in the application initializer instead of relying on
+implicit filesystem behavior.
+
+For a Rails app that serves sitemap files from `public/`:
+
+```ruby
+Indexmap.configure do |config|
+  config.base_url = -> { "https://example.com" }
+  config.storage = -> do
+    Indexmap::Storage::Filesystem.new(
+      path: Rails.public_path,
+      public_url: config.base_url
+    )
+  end
+end
+```
+
+For a production app that stores sitemap files elsewhere, point the storage
+adapter at that production location and keep `public_url` on the public origin
+where search engines fetch the files.
+
+```ruby
+Indexmap.configure do |config|
+  config.base_url = -> { "https://example.com" }
+  config.storage = -> do
+    Indexmap::Storage::Filesystem.new(
+      path: Rails.root.join("storage/sitemaps"),
+      public_url: config.base_url
+    )
+  end
+end
+```
+
+### 2. Keep Relative Storage Keys Stable
+
+If your app stores sitemap files under a directory, keep those directory
+prefixes in `config.index_filename`, section filenames, and named output
+filenames.
+
+```ruby
+Indexmap.configure do |config|
+  config.index_filename = "sitemaps/sitemap.xml"
+  config.sections = -> do
+    [
+      Indexmap::Section.new(
+        filename: "sitemaps/sitemap-pages.xml",
+        entries: Sitemap.pages
+      )
+    ]
+  end
+end
+```
+
+Do not work around older versions by stripping paths down to basenames. The
+parser, validator, pingers, and filesystem adapter now preserve relative storage
+keys such as `sitemaps/sitemap.xml`.
+
+### 3. Update Custom Storage Backends
+
+Custom storage backends must accept the documented `content_type:` keyword on
+`write`. The format task rewrites XML files and passes the XML content type.
+
+```ruby
+class SitemapStorage
+  def write(filename, body, content_type:)
+    # Persist body under filename and store content_type when the backend
+    # uses it.
+  end
+
+  def read(filename)
+  end
+
+  def exist?(filename)
+  end
+
+  def list(prefix:, suffix:)
+  end
+
+  def delete(filename)
+  end
+
+  def public_url(filename)
+  end
+end
+```
+
+### 4. Verify Development And Production
+
+Run the same lifecycle tasks against the storage configuration each environment
+will use.
+
+Development:
+
+```bash
+bin/rails indexmap:sitemap:create
+bin/rails indexmap:sitemap:validate
+bin/rails indexmap:sitemap:format
+```
+
+Production or staging:
+
+```bash
+RAILS_ENV=production bin/rails indexmap:sitemap:create
+RAILS_ENV=production bin/rails indexmap:sitemap:validate
+RAILS_ENV=production bin/rails indexmap:ping
+```
+
+Before deploying, confirm:
+
+- `config.base_url` is the public origin for the environment.
+- `config.index_filename` identifies the public root sitemap and exists in the
+  configured storage.
+- The root sitemap references every current child sitemap. Unreferenced sitemap
+  files in storage are intentionally ignored by the pingers.
+- `config.storage.public_url(config.index_filename)` returns the fetchable root
+  sitemap URL.
+- `indexmap:sitemap:validate` passes against the same storage used by
+  production.
+- If IndexNow is enabled, the `<key>.txt` verification file is present in the
+  configured storage or `config.index_now.key` is set.

data/docs/architecture.md

@@ -0,0 +1,55 @@
+# How Indexmap works
+
+> This describes unreleased main. See [release status](../README.md#release-status).
+
+The host supplies eligible URLs and content dates. Indexmap generates, validates,
+and publishes XML. A web server, CDN, or explicitly mounted Rails controller
+serves those artifacts. Crawlers do not trigger generation or query section sources.
+
+```mermaid
+flowchart TD
+  manifest["Host manifest: static entries or database enumeration"] --> writer
+  ruby["Indexmap.create(output)"] --> creator["Creator: selected output"]
+  task["Rake create task"] --> runner["TaskRunner#create: default output"]
+  job["Host refresh job"] --> runner
+  runner --> creator
+  creator --> writer["Writer: consume sources once and partition XML"]
+  writer --> staging["Temporary disk staging"]
+  staging --> validate["Validator: complete graph and disk-backed duplicate checks"]
+  validate --> mode{"Publication mode"}
+  mode -->|legacy| sequential["Write children, then stable root sequentially"]
+  mode -->|atomic| prepare["Prepare children outside the root lock"]
+  prepare --> atomic["Check revision and activate root under a short lock"]
+  sequential --> storage["Configured storage"]
+  atomic --> storage
+  creator -->|successful return to TaskRunner| hooks["IndexNow key handling, then after_create callbacks"]
+  storage --> static["Static server or CDN"]
+  storage --> controller["Opt-in Rails sitemap controller"]
+  static --> crawler["Crawler fetches published XML"]
+  controller --> crawler
+  config["Host robots policy"] --> robots["Static robots.txt or opt-in Rails robots controller"]
+  robots --> crawler
+  storage --> ping["Separate Google or IndexNow submission"]
+```
+
+Generation and validation errors occur before publication. Legacy writes can
+leave mixed files if publication itself fails. Atomic mode retains the previous
+root until activation succeeds; a competing winner causes
+`Indexmap::SupersededPublication`. Built-in adapters clean up ordinary failed
+builds; retention covers artifacts left by crashes. Cleanup errors are reported
+explicitly. Callbacks and key handling run after publication,
+so their failures do not undo it. Neither generation entrypoint submits to search
+engines automatically. A host can choose to submit from a callback, but separate
+submission makes refresh retries easier to operate.
+
+| Entrypoint | Purpose |
+| --- | --- |
+| `Indexmap.create(:name)` | Generate, validate, and publish one output; defaults to `:default`. |
+| `Indexmap::TaskRunner.new.create` | Generate the default output, handle the IndexNow key, then invoke callbacks. |
+| `indexmap:sitemap:create` | Run `TaskRunner#create` and print the writer's recorded counts. |
+| `Writer#each_file` / `Writer#write` | Render XML files; callers own validation and publication. `write` collects all files. |
+| `indexmap:sitemap:format` | Optionally pretty-print legacy stored files; atomic outputs reject mutation. |
+| Sitemap / robots controllers | Serve artifacts or configuration; never generate a sitemap. |
+
+See [domain terms](domain-terms.md), [deployment choices](deployment.md), and the
+[storage contract](storage.md).

data/docs/deployment.md

@@ -0,0 +1,223 @@
+# Choose a deployment path
+
+> This guide describes unreleased main, not RubyGems 0.8.2.
+> See [release status](../README.md#release-status) before installing.
+
+Source data and HTTP delivery are separate decisions. A database-backed sitemap
+can be served as a static file after a scheduled refresh. Static marketing URLs
+can be served by a Rails controller. Neither choice generates XML on crawler
+requests. Robots policy is independent of both.
+
+| Situation | Generate | Store and serve | Robots |
+| --- | --- | --- | --- |
+| Static marketing site | Build or release step from an explicit manifest; no database required by the source | Include `public/` artifacts in the deployed image/site | Static file, or Rails endpoint if environment policy varies |
+| Database content, one server | Refresh after database preparation and on a host schedule | Persistent filesystem shared by the refresh and web processes; static server or Rails | Static or dynamic policy |
+| Database content, multiple containers | Host job on GoodJob/Solid Queue or a release task | Shared persistent filesystem with working locks, or Active Storage with shared backing storage | Dynamic policy is useful for production-mode previews |
+| Development or staging | Run the same generation/validation lifecycle with test content | Local filesystem or isolated staging storage | Disallow crawling; do not submit to search engines |
+
+See the [pipeline diagram](architecture.md) for the generation/publication boundary.
+
+## Install and configure
+
+Follow the [Rails installation](../README.md#rails-usage), including
+`require: false` and `bin/rails generate indexmap:install`. Edit the configure
+block inside `Indexmap::HostConfiguration.apply`, keeping its idempotent wrapper.
+Construct storage once there when its settings are fixed:
+
+```ruby
+config.base_url = ENV.fetch("INDEXMAP_BASE_URL", "http://localhost:3000")
+config.storage = Indexmap::Storage::Filesystem.new(
+  path: Rails.public_path, public_url: config.base_url
+)
+```
+
+Always set `INDEXMAP_BASE_URL` to the actual public origin in deployment. For
+private filesystem storage change `path` to `Rails.root.join("storage/sitemaps")`
+and explicitly serve it through the Rails controller or web server. That directory
+does not become publicly reachable merely because it has a `public_url` setting.
+Use separate storage namespaces for development, staging, and production.
+
+### Static sources
+
+For a small, database-free manifest, add to the same configure block:
+
+```ruby
+config.format = :single_file
+config.entries = -> do
+  ["/", "/about", "/pricing"].map do |path|
+    Indexmap::Entry.new(loc: "#{config.base_url}#{path}")
+  end
+end
+```
+
+Use an origin without a trailing slash for this example. Supply actual content
+modification dates when available; omit `lastmod` when they are unknown. For
+larger or grouped manifests use `config.sections` and index mode as in the
+[README](../README.md#default-index-mode). Single-file mode fails if it needs to
+split; index mode partitions automatically.
+
+The gem does not require a database for a static source. The host Rails boot may
+still require services through unrelated initializers. Keep its static build
+entrypoint free of database queries, or generate through a plain Ruby script
+with `require "indexmap"`, `Indexmap.configure`, and `Indexmap.create`.
+
+### Database-backed sources
+
+Use the [database-backed guide](dynamic-sitemaps.md#configure-generation) for lazy
+eligible-record enumeration. Generate after migrations and source transactions
+commit. Refresh on a recurring schedule when eligibility depends on time, and
+on meaningful publishing events if needed. Hosts own queries, routes, meaningful
+dates, authorization context, jobs, and schedules.
+
+For independent static and database outputs, use
+[named outputs](../README.md#named-outputs). They do not automatically form a
+combined index. Advertise every independent public root in robots.txt and submit
+each separately if needed. Generate a deferred output successfully before
+advertising it.
+
+## Development verification
+
+```sh
+INDEXMAP_BASE_URL=http://localhost:3000 bin/rails indexmap:sitemap:create
+INDEXMAP_BASE_URL=http://localhost:3000 bin/rails indexmap:sitemap:validate
+bin/rails server
+```
+
+From another terminal, fetch the chosen endpoints:
+
+```sh
+curl -i http://localhost:3000/sitemap.xml
+curl -i http://localhost:3000/robots.txt
+```
+
+Expect XML with the configured localhost URLs and robots policy that disallows
+crawling. If the root is an index, fetch one referenced child too. Check HEAD
+and conditional responses when using controllers. Creation validates before
+publishing; the separate validate command checks the stored result. Formatting
+is optional for legacy output and unavailable for atomic publications.
+
+## Production static files
+
+For static manifests, generate with the final public origin during a build or
+release step, then include every generated root and child in the deployed
+artifacts:
+
+```sh
+RAILS_ENV=production INDEXMAP_BASE_URL=https://example.com \
+  bin/rails indexmap:sitemap:create
+RAILS_ENV=production INDEXMAP_BASE_URL=https://example.com \
+  bin/rails indexmap:sitemap:validate
+```
+
+Do not generate database content during an image build unless database access
+and snapshot semantics are deliberate. Use a post-migration release step or job
+instead. A release container's generated files must reach the serving instances;
+files left on its private filesystem are unavailable to web containers.
+
+Serve `public/` through the deployment's web server/CDN, or explicitly enable
+Rails static serving. If the host uses an environment flag, configure its
+production environment accordingly:
+
+```ruby
+config.public_file_server.enabled = ENV["RAILS_SERVE_STATIC_FILES"] == "1"
+```
+
+Set the flag only where Rails should serve these files. A reverse proxy or CDN
+can serve them without Rails static serving. Keep stable roots on a short TTL,
+and use a long immutable TTL only for atomic generated children. Legacy child
+names are overwritten and must not receive immutable caching. Keep previously
+referenced children available through the cached-root and rollback period.
+
+A complete image deployed to one server can be enough for a static manifest.
+Rolling deployments across different images can expose roots and children from
+different releases; use shared storage or deploy all artifacts before switching
+the root. Atomic mode protects publication in its configured store, not an
+arbitrary deployment system's separate file-copy step.
+
+## Production shared storage and Rails delivery
+
+For refresh jobs across containers, use shared persistent storage. Configure
+`config.atomic = true` for revision-checked activation and immutable children.
+Choose Filesystem only when locks and rename work across processes, or use the
+[Active Storage adapter](../README.md#storage) with a unique artifact filename
+index. A shared Active Storage database alone is insufficient if each container
+uses a different local disk service.
+
+Mount the root and generated child routes from
+[Serve published XML](dynamic-sitemaps.md#serve-published-xml). Store XML outside
+`public/` when the controller should own delivery, and remove old static files
+that would shadow its routes. The shared controller serves configured roots and
+atomic children; legacy index children require deliberate static or host delivery.
+A single-file output needs only its root route.
+
+Use one host refresh owner and the host's actual queue adapter; see
+[background generation](../README.md#background-generation). Apply the generated
+configuration in jobs. `TaskRunner#create` includes key-file handling and
+callbacks; `Indexmap.create(:name)` only generates an output. Callbacks can fail
+after files are published, so make them safe to repeat and monitor job failures.
+
+Schedule pruning separately for atomic output:
+
+```sh
+RAILS_ENV=production RETENTION_HOURS=48 bin/rails indexmap:sitemap:prune
+```
+
+Supply the same storage/origin settings as generation. Retain children longer
+than cached roots, stale responses, and rollback windows can reference them.
+Replace old callbacks that delete files absent from configured section names.
+
+## Static or dynamic robots.txt
+
+Choose one owner for `/robots.txt`:
+
+- **Static:** manage `public/robots.txt` or a web-server file. For a public
+  production site, it can contain the following. Deploy a separate disallow file
+  for development, staging, and previews.
+- **Dynamic:** mount the [robots controller](../README.md#dynamic-robots), remove
+  `public/robots.txt`, and configure an explicit environment predicate. It works
+  with static or controller-served sitemap XML and does not read sitemap storage.
+
+Public production file:
+
+```text
+User-agent: *
+Allow: /
+Sitemap: https://example.com/sitemap.xml
+```
+
+Development, staging, or preview file:
+
+```text
+User-agent: *
+Disallow: /
+```
+
+Dynamic policy inside the configure block:
+
+```ruby
+config.robots.indexable = -> { Rails.env.production? && ENV["PUBLIC_INDEXING"] == "1" }
+config.robots.groups = [{user_agent: "*", allow: ["/"], disallow: ["/admin/", "/search"]}]
+```
+
+Set `PUBLIC_INDEXING=1` only on the intended public deployment. A preview can
+run with `RAILS_ENV=production`; that alone must not enable crawling. When the
+predicate is false, Indexmap emits disallow-all without sitemap advertisements.
+Static robots files ignore this predicate. Keep host-specific exclusions in the
+chosen policy. Robots instructions are not access control and do not ensure a
+page is removed from an index; protect private routes independently.
+
+## Verify the deployed result
+
+Fetch the actual public root, a referenced child (in index mode), and robots.txt.
+Check content, content types, redirects, cache headers, and intended allow/disallow
+rules from outside the container. Confirm that job writes are visible from every
+serving instance and that a refresh failure retains the old root in atomic mode.
+A storage validation pass does not prove HTTP delivery.
+
+Keep search-engine submission separate. Test with
+`INDEXNOW_DRY_RUN=1 bin/rails indexmap:index_now:ping` against the correct
+environment before live submission; a dry run still needs an IndexNow key.
+[Provision the verification key](search-engine-ping.md) at its public URL,
+especially when XML lives outside `public/`. Do not submit development or staging
+URLs. Google credentials and property setup, IndexNow selection previews, and
+attempt reports are covered in the [search-engine guide](search-engine-ping.md).

data/docs/domain-terms.md

@@ -0,0 +1,27 @@
+# Domain terms
+
+| Term | Meaning |
+| --- | --- |
+| Configuration | `Indexmap::Configuration` holds gem settings, sources, outputs, callbacks, and robots policy. |
+| Configuration loader | Generated host module `Indexmap::HostConfiguration.apply`; applies the settings once per process. It is separate from the gem's configuration object. |
+| Output | One independently generated target with a stable root filename. Named outputs share storage and are not automatically linked into the default output. |
+| Section | A host-defined filename and enumerable of page entries, used in index mode. |
+| Page entry | `Indexmap::Entry`: the host's `loc` and optional meaningful content `lastmod`. |
+| Parsed entry | `Indexmap::Parser::Entry`: `loc`, `lastmod`, and the source sitemap key/URL. Returned by parsing and validation. |
+| Partition | A bounded child XML document produced from a section. One section may produce several partitions. |
+| Root | The configured `index_filename`: an index of children, or a URL set in single-file mode. |
+| Publication | One generated output made visible in storage. Atomic publications have immutable children and a guarded root activation. |
+| Generation | The unique identifier used for an atomic build's child keys and retained root archive. |
+| Artifact | A stored XML file, including a root, child, or retained root archive. `Indexmap::Artifact` is the writer's in-memory filename/body/content-type value, not Ruby's filesystem `::File`. |
+| Storage key | A relative filename, possibly including directories, such as `sitemaps/sitemap.xml`. It is separate from its public HTTP URL. |
+| Static source | Explicit routes or manifest entries that can be generated without querying a database. |
+| Dynamic source | Host data selected at refresh time, often through a lazy database enumeration. It does not mean generation during HTTP requests. |
+| Format | `:index` or `:single_file` selects XML shape. The separate `format` task changes whitespace in legacy artifacts. |
+| Robots policy | Crawl instructions served as `/robots.txt`. It is independent of sitemap storage and refresh scheduling. |
+
+Use these terms in host manifests and operational instructions. The generated
+host loader is `Indexmap::HostConfiguration`; `Indexmap::Configuration` is the
+gem's settings object. `Indexmap::Artifact` is rendered data and `::File` is Ruby's
+filesystem class. The former `IndexmapConfiguration` and `Indexmap::Storage::File`
+names are replaced by this unreleased contract change; see
+[UPGRADE.md](../UPGRADE.md#host-loader-and-rendered-artifacts) for migration.