indexmap 0.8.2 → 0.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (54) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +85 -0
  3. data/README.md +281 -94
  4. data/UPGRADE.md +361 -0
  5. data/docs/architecture.md +55 -0
  6. data/docs/deployment.md +223 -0
  7. data/docs/domain-terms.md +27 -0
  8. data/docs/dynamic-sitemaps.md +302 -0
  9. data/docs/search-engine-ping.md +368 -0
  10. data/docs/storage.md +94 -0
  11. data/lib/generators/indexmap/install_generator.rb +1 -1
  12. data/lib/generators/indexmap/templates/initializer.rb.tt +18 -18
  13. data/lib/indexmap/artifact.rb +9 -0
  14. data/lib/indexmap/configuration.rb +12 -1
  15. data/lib/indexmap/creator.rb +45 -42
  16. data/lib/indexmap/index_now_configuration.rb +3 -1
  17. data/lib/indexmap/output.rb +19 -5
  18. data/lib/indexmap/parser.rb +112 -53
  19. data/lib/indexmap/pinger/base.rb +6 -5
  20. data/lib/indexmap/pinger/google.rb +12 -8
  21. data/lib/indexmap/pinger/index_now.rb +97 -52
  22. data/lib/indexmap/publication.rb +89 -0
  23. data/lib/indexmap/rails.rb +6 -0
  24. data/lib/indexmap/railtie.rb +7 -0
  25. data/lib/indexmap/robots.rb +67 -0
  26. data/lib/indexmap/robots_controller.rb +13 -0
  27. data/lib/indexmap/sitemaps_controller.rb +57 -0
  28. data/lib/indexmap/storage/active_storage.rb +132 -1
  29. data/lib/indexmap/storage/filesystem.rb +182 -11
  30. data/lib/indexmap/storage/memory.rb +1 -1
  31. data/lib/indexmap/task_runner.rb +21 -28
  32. data/lib/indexmap/validator/duplicates.rb +83 -0
  33. data/lib/indexmap/validator.rb +171 -59
  34. data/lib/indexmap/version.rb +1 -1
  35. data/lib/indexmap/writer.rb +89 -34
  36. data/lib/indexmap.rb +14 -1
  37. data/lib/tasks/indexmap_tasks.rake +21 -2
  38. data/test/indexmap/correctness_test.rb +153 -0
  39. data/test/indexmap/install_generator_test.rb +36 -4
  40. data/test/indexmap/parser_test.rb +77 -0
  41. data/test/indexmap/pinger/google_test.rb +22 -0
  42. data/test/indexmap/pinger/index_now_test.rb +168 -0
  43. data/test/indexmap/publication_test.rb +280 -0
  44. data/test/indexmap/rails_test.rb +22 -0
  45. data/test/indexmap/robots_test.rb +73 -0
  46. data/test/indexmap/sitemaps_controller_test.rb +50 -0
  47. data/test/indexmap/storage_test.rb +27 -0
  48. data/test/indexmap/task_runner_test.rb +24 -0
  49. data/test/indexmap/validator_test.rb +14 -0
  50. data/test/indexmap/writer_test.rb +42 -0
  51. data/test/integration/queue_checks.rb +244 -0
  52. metadata +33 -51
  53. data/lib/indexmap/storage/file.rb +0 -11
  54. data/test/release_task_test.rb +0 -86
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: d3d31112daf5ee0b501338d564a4e7e9ef30487c88e65439a67543faac070d64
4
- data.tar.gz: 550713d5205e26bc9c42a4a37660d0c72e493898b47f7bf5dbb4ef92e2488e6b
3
+ metadata.gz: 0a88a19e175e22156b0b054041e4d673365eee9089e5fc42347a31f1583dc2e9
4
+ data.tar.gz: 1404999d7723a994cdf5763715dc61c6366a435dace94d9f0796bd82aad69753
5
5
  SHA512:
6
- metadata.gz: 10da855e190aab04ccd1e3dcc4fe5e49125bd431eb9831c9ea0e983169f11c08d0ce1beb093bd16d8576d637b850fa3d8da74c013b65cd5982bab0452097d272
7
- data.tar.gz: 6e4c5cdd0a014d1e897ba94d79f22f1acb69f0eb6ff1ff4f23ef756fb1884b1ef27df96f5181b665bdefd9e80f4925b7fe985a1ddc51bdd91e1f9d9e47af98af
6
+ metadata.gz: e610924028d149f000c15f5d0c570232457040aa01b77769c271fe94d4ecc9da4ce067a78167b9c659b892c75f923e6d5b0861838ed0ff7563d7c63753c24673
7
+ data.tar.gz: 5c4986f19bcd67274948b9c4c9a6db4c8a1b7855f3b3c20774179fdc13ee0c300df4c10bf79249d42f358faf4636ea0ba60788eb692fe055a5a987814c6ddfc6
data/CHANGELOG.md CHANGED
@@ -5,6 +5,91 @@ All notable changes to this project will be documented in this file.
5
5
  The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
6
6
  and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
7
 
8
+ ## [0.10.0] - 2026-11-04
9
+
10
+
11
+ ### Added
12
+
13
+ - require Ruby 4 and adopt Gemkit
14
+
15
+ - require Ruby 4 and adopt Gemkit ([#27](https://github.com/ethos-link/indexmap/pull/27))
16
+
17
+
18
+
19
+
20
+ ### Fixed
21
+
22
+ - push the version tag during preparation
23
+
24
+ - push the version tag during preparation ([#26](https://github.com/ethos-link/indexmap/pull/26))
25
+
26
+
27
+
28
+ ## [0.9.0] - 2026-11-04
29
+
30
+
31
+ ### Added
32
+
33
+ - validate and expose public crawler output
34
+
35
+ - validate and expose public crawler output ([#21](https://github.com/ethos-link/indexmap/pull/21))
36
+
37
+ - stream and atomically publish dynamic inventories
38
+
39
+ - stream and atomically publish dynamic inventories ([#23](https://github.com/ethos-link/indexmap/pull/23))
40
+
41
+
42
+
43
+
44
+ ### Changed
45
+
46
+ - reconcile writer branch with merged sitemap fixes
47
+
48
+ - synchronize publication branch with resolved writer base
49
+
50
+
51
+
52
+
53
+ ### Documentation
54
+
55
+ - explain background sitemap generation ([#19](https://github.com/ethos-link/indexmap/pull/19))
56
+
57
+ - explain public key setup in Rails ([#20](https://github.com/ethos-link/indexmap/pull/20))
58
+
59
+ - clarify usage limits and implementation status
60
+
61
+ - clarify retries are disabled by default
62
+
63
+ - document unreleased APIs and sitemap deployment choices
64
+
65
+
66
+
67
+
68
+ ### Fixed
69
+
70
+ - consume sitemap section entries once
71
+
72
+ - consume sitemap section entries once ([#22](https://github.com/ethos-link/indexmap/pull/22))
73
+
74
+ - separate review preparation from release tagging
75
+
76
+ - prepare and push releases from main
77
+
78
+ - install git-cliff before running release tests
79
+
80
+ - prepare and push releases from main ([#25](https://github.com/ethos-link/indexmap/pull/25))
81
+
82
+
83
+
84
+
85
+ ### fix
86
+
87
+ - harden publication and Rails task loading
88
+
89
+ - harden sitemap publication and Rails integration ([#24](https://github.com/ethos-link/indexmap/pull/24))
90
+
91
+
92
+
8
93
  ## [0.8.2] - 2026-07-22
9
94
 
10
95
 
data/README.md CHANGED
@@ -5,7 +5,7 @@
5
5
 
6
6
  `indexmap` is a small Ruby gem for generating XML sitemaps from explicit Ruby data.
7
7
 
8
- It is designed for Rails apps that want:
8
+ Use it from Ruby or an optional Rails 8 integration for:
9
9
 
10
10
  - deterministic sitemap output
11
11
  - plain Ruby configuration
@@ -14,6 +14,9 @@ It is designed for Rails apps that want:
14
14
 
15
15
  By default, `indexmap` writes a sitemap index plus one or more child sitemap files. For simpler sites, it also supports `:single_file` mode, which writes a single `urlset` directly to `sitemap.xml`.
16
16
 
17
+ See [CHANGELOG.md](CHANGELOG.md) for published changes and [UPGRADE.md](UPGRADE.md)
18
+ for host migration steps.
19
+
17
20
  ## Installation
18
21
 
19
22
  Add this line to your application's Gemfile:
@@ -44,6 +47,14 @@ directory prefix such as `sitemaps/`.
44
47
 
45
48
  ## Documentation
46
49
 
50
+ - [How Indexmap works](docs/architecture.md): pipeline diagram and entrypoints.
51
+ - [Domain terms](docs/domain-terms.md): sections, partitions, outputs, publications,
52
+ roots, and entry types.
53
+ - [Deployment guide](docs/deployment.md): development and production, static and
54
+ database-backed sources, static files or Rails delivery, and robots.txt.
55
+ - [Storage contract](docs/storage.md): legacy, streaming, atomic, and pruning APIs.
56
+ - [Dynamic sitemaps](docs/dynamic-sitemaps.md): streaming and partitions, atomic
57
+ publication, Rails delivery, retention, jobs, and host adoption.
47
58
  - [Search engine ping](docs/search-engine-ping.md): Google Search Console
48
59
  credentials, Google property configuration, and IndexNow key setup.
49
60
  - [UPGRADE.md](UPGRADE.md): host-app migration steps for public contract
@@ -64,10 +75,15 @@ sections = [
64
75
  )
65
76
  ]
66
77
 
67
- Indexmap::Writer.new(
68
- sections: sections,
69
- base_url: "https://example.com"
70
- ).write
78
+ Indexmap.configure do |config|
79
+ config.base_url = "https://example.com"
80
+ config.storage = Indexmap::Storage::Filesystem.new(
81
+ path: "public", public_url: config.base_url
82
+ )
83
+ config.sections = sections
84
+ end
85
+
86
+ Indexmap.create
71
87
  ```
72
88
 
73
89
  ## Rails Usage
@@ -79,7 +95,7 @@ bin/rails generate indexmap:install
79
95
  ```
80
96
 
81
97
  The generator creates `config/initializers/indexmap.rb` with an idempotent
82
- `IndexmapConfiguration.apply` loader and adds conditional `indexmap/tasks`
98
+ `Indexmap::HostConfiguration.apply` loader and adds conditional `indexmap/tasks`
83
99
  loading to the application's `Rakefile`. This preserves `bin/rails --tasks` and
84
100
  the `indexmap:*` tasks without loading Indexmap during normal Rails boot.
85
101
 
@@ -88,12 +104,10 @@ Customize the `Indexmap.configure` block inside the generated initializer:
88
104
  ```ruby
89
105
  Indexmap.configure do |config|
90
106
  config.base_url = -> { "https://example.com" }
91
- config.storage = -> do
92
- Indexmap::Storage::Filesystem.new(
93
- path: Rails.public_path,
94
- public_url: config.base_url
95
- )
96
- end
107
+ config.storage = Indexmap::Storage::Filesystem.new(
108
+ path: Rails.public_path,
109
+ public_url: config.base_url
110
+ )
97
111
  config.sections = -> do
98
112
  [
99
113
  Indexmap::Section.new(
@@ -111,7 +125,7 @@ Runtime callers and jobs must apply the generated configuration before using
111
125
  Indexmap:
112
126
 
113
127
  ```ruby
114
- IndexmapConfiguration.apply
128
+ Indexmap::HostConfiguration.apply
115
129
  Indexmap.create
116
130
  ```
117
131
 
@@ -122,14 +136,188 @@ Then run:
122
136
 
123
137
  ```bash
124
138
  bin/rails indexmap:sitemap:create
125
- bin/rails indexmap:sitemap:format
126
139
  bin/rails indexmap:sitemap:validate
127
140
  ```
128
141
 
129
- `indexmap:sitemap:create` is the main task. It builds sitemap files in memory,
130
- formats them, validates the result, then writes the final XML files to the
131
- configured storage. Existing sitemap files are left untouched if generation or
132
- validation fails.
142
+ Creation already validates. `indexmap:sitemap:format` optionally pretty-prints
143
+ legacy XML after publication; it rejects atomic outputs.
144
+
145
+ `indexmap:sitemap:create` is the main task. It stages partitioned XML on temporary
146
+ disk, validates the complete result, then publishes to configured storage.
147
+ Generation and validation failures leave published files untouched. Legacy
148
+ publication remains sequential; enable atomic mode for immutable children and
149
+ revision-checked root activation. See [dynamic sitemaps](docs/dynamic-sitemaps.md).
150
+
151
+ ### Background Generation
152
+
153
+ Keep the refresh job and its scheduling in the host application. Indexmap does
154
+ not select a queue adapter, run workers, or deduplicate jobs. Apply the generated
155
+ configuration inside the job. Use `Indexmap::TaskRunner.new.create` for the
156
+ rake task's complete lifecycle, including `after_create` callbacks and IndexNow
157
+ key-file handling; `Indexmap.create` only generates the selected output.
158
+
159
+ #### GoodJob
160
+
161
+ With GoodJob 4.22.1 or newer, use its public labelled concurrency rules:
162
+
163
+ ```ruby
164
+ class SitemapRefreshJob < ApplicationJob
165
+ include GoodJob::ActiveJobExtensions::Concurrency
166
+
167
+ good_job_concurrency_rule(
168
+ perform_limit: 1,
169
+ label: "sitemap-refresh",
170
+ apply_label: true
171
+ )
172
+
173
+ good_job_concurrency_rule(
174
+ total_limit: 1,
175
+ label: -> { "sitemap-refresh:#{arguments.first}" },
176
+ apply_label: true
177
+ )
178
+
179
+ def perform(_version)
180
+ Indexmap::HostConfiguration.apply
181
+ Indexmap::TaskRunner.new.create
182
+ end
183
+ end
184
+ ```
185
+
186
+ Enqueue with a release identifier: `SitemapRefreshJob.perform_later(version)`.
187
+ The version label limits unfinished work for that release; the shared label
188
+ limits generation across releases to one execution at a time. A newer release
189
+ can enqueue while an older refresh is unfinished. A repeat enqueue is allowed
190
+ after completion: these controls do not provide permanent once-per-release
191
+ tracking. Use `perform_later`; direct `perform_now` calls bypass GoodJob's limits.
192
+ See [GoodJob concurrency controls](https://github.com/bensheldon/good_job#concurrency-controls).
193
+
194
+ #### Solid Queue
195
+
196
+ With Solid Queue 1.7 or newer, serialize writes to the shared sitemap output:
197
+
198
+ ```ruby
199
+ class SitemapRefreshJob < ApplicationJob
200
+ limits_concurrency to: 1, key: "sitemap-refresh",
201
+ duration: 30.minutes, on_conflict: :block
202
+
203
+ def perform(_version)
204
+ Indexmap::HostConfiguration.apply
205
+ Indexmap::TaskRunner.new.create
206
+ end
207
+ end
208
+ ```
209
+
210
+ Choose a duration longer than the expected generation time. `:block` retains
211
+ every refresh and runs them serially; it does not suppress duplicate enqueues.
212
+ Using `on_conflict: :discard` coalesces conflicting requests, including requests
213
+ for newer releases when the key is shared. A version-based key such as
214
+ `->(version) { "sitemap-refresh:#{version}" }` scopes conflicts to that release,
215
+ but permits different releases to write concurrently. Select that tradeoff
216
+ explicitly; it is not equivalent to GoodJob's two rules above. Direct
217
+ `perform_now` calls bypass queue coordination.
218
+ See [Solid Queue concurrency controls](https://github.com/rails/solid_queue#concurrency-controls).
219
+
220
+ #### Triggering Refreshes
221
+
222
+ Enqueue from one release task, a recurring schedule, or application startup.
223
+ Rails' `after_initialize` hook can enqueue from a worker application without
224
+ depending on a backend's worker callbacks:
225
+
226
+ ```ruby
227
+ Rails.application.config.after_initialize do
228
+ if Rails.env.production? && ENV["PROCESS_ROLE"] == "worker"
229
+ SitemapRefreshJob.perform_later(ENV.fetch("RELEASE_VERSION")) do |job|
230
+ raise job.enqueue_error if job.enqueue_error
231
+ end
232
+ end
233
+ end
234
+ ```
235
+
236
+ `PROCESS_ROLE` and `RELEASE_VERSION` are application-defined environment values.
237
+ Configure them in the deployment, with the worker role flag limited to worker
238
+ processes. `after_initialize` runs for every Rails process, including commands;
239
+ it indicates application initialization, not worker readiness. Complete database
240
+ preparation before starting workers. Suppressed enqueues are expected; report
241
+ actual enqueue errors. Avoid scheduling the same refresh from both startup and
242
+ a release task.
243
+
244
+ Use storage shared by all serving containers, such as object storage through
245
+ the Active Storage adapter. A job container's local filesystem is not shared
246
+ with web containers unless the deployment mounts the same persistent storage.
247
+
248
+ #### Operating Practices And Current Limits
249
+
250
+ Generate database-backed content after database preparation, through one
251
+ host-owned refresh job. Load the existing configuration inside that job and
252
+ query eligible records when it runs. Pass release or output identifiers rather
253
+ than relations, URL inventories, or credentials. Keep queue selection, recurring
254
+ schedules, and any application authorization context in the host.
255
+
256
+ Use a shared concurrency key or label for jobs writing the same destination,
257
+ across releases. Prefer blocking or rescheduling conflicts to discarding a newer
258
+ refresh. Backend controls do not protect manual rake invocations, and Solid
259
+ Queue's concurrency duration can expire before an unusually long job finishes.
260
+ Avoid concurrent writers through all entrypoints in legacy mode: an interrupted
261
+ sequential publication can leave mixed generations. Opt-in atomic mode checks
262
+ root revisions under a storage lock and retains previous generations. See the
263
+ [publication and retention contract](docs/dynamic-sitemaps.md#atomic-publication-and-storage).
264
+
265
+ Keep generation and search-engine submission separate. A refresh retry should
266
+ not automatically resend successful IndexNow batches. Make `after_create`
267
+ callbacks safe to repeat: they run after publication, and a callback failure does
268
+ not roll published files back. Validate the deployed sitemap and preview IndexNow
269
+ selection before submitting. See [selection and attempt reports](docs/search-engine-ping.md#inspect-selection-and-actual-attempts).
270
+
271
+ Batch source queries with native Rails APIs and select the fields needed for
272
+ URLs and dates. Supply meaningful content dates; do not use the refresh time as
273
+ `lastmod`. Each sitemap must stay within 50,000 entries and 52,428,800 uncompressed
274
+ bytes. Index mode partitions automatically; single-file mode rejects an output
275
+ that needs splitting. Measure the complete refresh before adding parallel jobs or
276
+ caches. Supply lazy sources to benefit from incremental generation and validation;
277
+ app-owned arrays and explicit inventory-returning APIs still retain their data.
278
+
279
+ Generation uses the same synchronous entrypoints with GoodJob and Solid Queue;
280
+ the examples above describe their different concurrency behavior. Verify the
281
+ host's actual adapter, workers, shared storage, failure reporting, and intended
282
+ system context in integration tests. Gem tests alone do not establish those
283
+ deployment guarantees.
284
+
285
+ ### Dynamic Robots
286
+
287
+ Indexmap can serve `/robots.txt` from configuration without querying content or
288
+ reading sitemap storage. Crawling defaults to disabled. Inside the generated
289
+ `Indexmap.configure` block, configure the site's policy:
290
+
291
+ ```ruby
292
+ config.robots.indexable = -> { Rails.env.production? && ENV["PUBLIC_INDEXING"] == "1" }
293
+ config.robots.groups = [
294
+ {user_agent: "*", allow: ["/"], disallow: ["/admin/", "/search"]}
295
+ ]
296
+ ```
297
+
298
+ The sitemap URL defaults to `base_url` plus `index_filename`. Override it with
299
+ `config.robots.sitemaps = ["https://example.com/sitemap.xml"]` when needed.
300
+ Groups and sitemap lists also accept callables. Keep them configuration-only;
301
+ do not query models in these callables.
302
+
303
+ In the application's `config/routes.rb`, load and mount the endpoint explicitly:
304
+
305
+ ```ruby
306
+ require "indexmap/robots_controller"
307
+ Indexmap::HostConfiguration.apply
308
+
309
+ Rails.application.routes.draw do
310
+ get "/robots.txt", to: Indexmap::RobotsController.action(:show)
311
+ end
312
+ ```
313
+
314
+ If the configuration loader lives outside an initializer, require that file
315
+ before applying it. Remove `public/robots.txt` so it cannot shadow the route.
316
+ The controller supports GET, HEAD, and conditional requests, uses a one-hour
317
+ public cache, and does not inherit application authentication or browser gates.
318
+ Set `PUBLIC_INDEXING` only for the intended public deployment; production-mode
319
+ previews should remain disabled. Robots rules do not replace authorization or
320
+ page-level indexing directives.
133
321
 
134
322
  ### Default Index Mode
135
323
 
@@ -138,6 +326,24 @@ This is the default behavior. `indexmap` writes:
138
326
  - `sitemap.xml` as a sitemap index
139
327
  - one or more child sitemap files from `config.sections`
140
328
 
329
+ Section entries can be enumerables. Indexmap consumes each section once and
330
+ calculates its index `lastmod` in that same pass. For database sources, return a
331
+ fresh enumerable from the configured sections callable on every generation:
332
+
333
+ ```ruby
334
+ config.sections = -> do
335
+ entries = PublicPage.indexable.find_each.lazy.map do |page|
336
+ Indexmap::Entry.new(loc: page.canonical_url, lastmod: page.updated_at)
337
+ end
338
+ [Indexmap::Section.new(filename: "sitemap-pages.xml", entries: entries)]
339
+ end
340
+ ```
341
+
342
+ `PublicPage`, its eligibility scope, and canonical URL method are host-app
343
+ examples. Keep queries and routing decisions in the host. Generation stages
344
+ bounded partitions and validates incrementally, using temporary disk for global
345
+ duplicate checks. See [memory and disk limits](docs/dynamic-sitemaps.md#configure-generation).
346
+
141
347
  ### Single-File Mode
142
348
 
143
349
  For sites that only want one `sitemap.xml` file:
@@ -145,7 +351,7 @@ For sites that only want one `sitemap.xml` file:
145
351
  ```ruby
146
352
  Indexmap.configure do |config|
147
353
  config.base_url = -> { "https://example.com" }
148
- config.storage = -> { Indexmap::Storage::Filesystem.new(path: Rails.public_path, public_url: config.base_url) }
354
+ config.storage = Indexmap::Storage::Filesystem.new(path: Rails.public_path, public_url: config.base_url)
149
355
  config.format = :single_file
150
356
  config.entries = -> do
151
357
  [
@@ -168,7 +374,7 @@ outputs write through the same configured storage as the default output.
168
374
  ```ruby
169
375
  Indexmap.configure do |config|
170
376
  config.base_url = -> { "https://example.com" }
171
- config.storage = -> { Indexmap::Storage::Filesystem.new(path: Rails.root.join("storage/sitemaps"), public_url: config.base_url) }
377
+ config.storage = Indexmap::Storage::Filesystem.new(path: Rails.root.join("storage/sitemaps"), public_url: config.base_url)
172
378
  config.sections = -> { Sitemap.sections }
173
379
 
174
380
  config.output :insights_data do |output|
@@ -194,8 +400,10 @@ Indexmap.create(:insights_data)
194
400
  Named outputs inherit `base_url` and `format` from the main configuration unless
195
401
  you override them. Storage is configured once and shared by every output.
196
402
 
197
- `Indexmap.create` uses the same safe publish flow as the rake task: build,
198
- format, validate, and then write the final XML file or files to storage.
403
+ `Indexmap.create` stages XML, validates it, then publishes the selected output.
404
+ Only `Indexmap::TaskRunner#create` (also used by the create task) handles the
405
+ IndexNow key file and `after_create` callbacks. Formatting is a separate optional
406
+ operation for legacy outputs; atomic outputs must be regenerated.
199
407
 
200
408
  ### Storage
201
409
 
@@ -243,17 +451,26 @@ storage.delete(filename)
243
451
  storage.public_url(filename)
244
452
  ```
245
453
 
454
+ Atomic publication, pruning, and streamed HTTP delivery require additional
455
+ capabilities. See the [storage contract](docs/storage.md) for signatures,
456
+ return values, and adapter requirements. Each operation instance resolves a
457
+ storage callable once and shares the adapter with its collaborators. Create a
458
+ fresh runner/pinger after changing destinations or tenant context. Direct reads
459
+ of `config.storage` still resolve the callable; construct an adapter once in the
460
+ host loader when its settings are fixed for the process.
461
+
246
462
  ### Deferred Dynamic Sections
247
463
 
248
464
  Use `after_create` when `indexmap:sitemap:create` should publish the default
249
465
  sitemap first, then schedule slower dynamic sections for the background. The
250
- callback runs only after the generated files have been formatted, validated, and
251
- replaced successfully.
466
+ callback runs only after the default output has been validated and published
467
+ and configured IndexNow key-file handling has succeeded. It also runs for
468
+ `Indexmap::TaskRunner#create`; `Indexmap.create` does not run callbacks.
252
469
 
253
470
  ```ruby
254
471
  Indexmap.configure do |config|
255
472
  config.base_url = -> { "https://example.com" }
256
- config.storage = -> { Indexmap::Storage::Filesystem.new(path: Rails.root.join("storage/sitemaps"), public_url: config.base_url) }
473
+ config.storage = Indexmap::Storage::Filesystem.new(path: Rails.root.join("storage/sitemaps"), public_url: config.base_url)
257
474
  config.sections = -> { Sitemap.sections }
258
475
 
259
476
  config.output :insights_data do |output|
@@ -273,13 +490,17 @@ Then the job can stay small:
273
490
  ```ruby
274
491
  class Insights::SitemapRefreshJob < ApplicationJob
275
492
  def perform
493
+ Indexmap::HostConfiguration.apply
276
494
  Indexmap.create(:insights_data)
277
495
  end
278
496
  end
279
497
  ```
280
498
 
281
- This keeps deploys fast: the deploy only waits for `indexmap:sitemap:create`,
282
- while database-dependent output is refreshed by the job backend.
499
+ The default output and named output are independent roots. List both public
500
+ root URLs in `config.robots.sitemaps` (or the static robots file) and submit each
501
+ separately if needed; named outputs are not automatically linked into the default
502
+ index. Publish the named output before advertising its URL. A deferred job can
503
+ fail after the default root is published, so monitor it separately.
283
504
 
284
505
  ## Validation And Parsing
285
506
 
@@ -312,6 +533,11 @@ The built-in validator checks for:
312
533
  See [Search engine ping](docs/search-engine-ping.md) for Google credential
313
534
  setup, Search Console property configuration, and IndexNow key provisioning.
314
535
 
536
+ For Rails apps with sitemaps outside `public/`, see
537
+ [serving an IndexNow key from `public/`](docs/search-engine-ping.md#rails-public-key-file-with-separate-sitemap-storage).
538
+ Key discovery searches the configured sitemap storage, so an existing public
539
+ key file needs explicit configuration in that setup.
540
+
315
541
  Available rake tasks:
316
542
 
317
543
  ```bash
@@ -326,84 +552,45 @@ If Google credentials are blank, `indexmap:google:ping` skips Google
326
552
  submission. If neither a configured IndexNow key nor a valid key file is
327
553
  present, `indexmap:index_now:ping` skips IndexNow submission.
328
554
 
329
- ## Development
330
-
331
- Run tests:
332
-
333
- ```bash
334
- bundle exec rake test
335
- ```
336
-
337
- Run lint:
338
-
339
- ```bash
340
- bundle exec rake standard
341
- ```
342
-
343
- Run the full default task:
344
-
345
- ```bash
346
- bundle exec rake
347
- ```
348
-
349
- Tests generate a coverage report automatically.
350
-
351
- Note: `Gemfile.lock` is intentionally not tracked for this gem, following normal Ruby library conventions.
555
+ ## License
352
556
 
353
- ### Git hooks
557
+ MIT License, see [LICENSE.txt](LICENSE.txt)
354
558
 
355
- We use [lefthook](https://lefthook.dev/) with the Ruby
356
- [commitlint](https://github.com/arandilopez/commitlint) gem to enforce
357
- Conventional Commits on every commit. We also use
358
- [Standard Ruby](https://standardrb.com/) to keep code style consistent. CI
359
- validates commit messages, Standard Ruby, tests, and git-cliff changelog
360
- generation on pull requests and pushes to main/master.
559
+ ## Development and releases
361
560
 
362
- Run the hook installer once per clone:
561
+ Shared tooling uses published [Ethos Gemkit](https://github.com/ethos-link/ethos-gemkit).
562
+ Keep the exact Gemfile version and workflow commit pins on the same release.
363
563
 
364
- ```bash
564
+ ```sh
565
+ bundle install
365
566
  bundle exec lefthook install
567
+ bundle exec rake verify
366
568
  ```
367
569
 
368
- ## Release
369
-
370
- Releases are tag-driven and published by GitHub Actions to RubyGems.
371
- Local release commands never publish directly.
372
-
373
- Install [git-cliff](https://git-cliff.org/) locally before preparing a
374
- release. The release task prepends the next `CHANGELOG.md` section from
375
- Conventional Commits.
570
+ The PostgreSQL integration suite covers Active Storage, GoodJob and Solid Queue:
376
571
 
377
- Before preparing a release, make sure you are on `main` or `master` with a
378
- clean worktree. If the release contains a breaking public-contract change,
379
- update `UPGRADE.md` with the host-app migration steps first.
380
-
381
- Then run one of:
382
-
383
- ```bash
384
- bundle exec rake 'release:prepare[patch]'
385
- bundle exec rake 'release:prepare[minor]'
386
- bundle exec rake 'release:prepare[major]'
387
- bundle exec rake 'release:prepare[0.1.0]'
572
+ ```sh
573
+ BUNDLE_GEMFILE=gemfiles/integration.gemfile bundle install
574
+ INDEXMAP_DATABASE_URL=postgresql://postgres@localhost:5432/indexmap_test \
575
+ BUNDLE_GEMFILE=gemfiles/integration.gemfile bundle exec ruby script/integration
388
576
  ```
389
577
 
390
- The task will:
578
+ Use a disposable `indexmap_test` database; the integration suite recreates it.
391
579
 
392
- 1. Prepend the next `CHANGELOG.md` section with `git-cliff`.
393
- 1. Update `lib/indexmap/version.rb`.
394
- 1. Commit the release changes.
395
- 1. Create and push the `vX.Y.Z` tag.
396
-
397
- The `Release` workflow then runs tests, publishes the gem to RubyGems,
398
- and creates the GitHub release from the changelog entry.
399
-
400
- ## License
401
-
402
- MIT License, see [LICENSE.txt](LICENSE.txt)
580
+ From clean, current `main`, run `bundle exec rake 'release:prepare[patch]'`
581
+ (or `minor`, `major`, or an explicit version). Review and merge the generated
582
+ release PR to publish through Actions. Shared prerequisites, registry credentials
583
+ and failed-job recovery are in the [Gemkit release guide](https://github.com/ethos-link/ethos-gemkit#releasing-a-gem).
403
584
 
404
- ## About
585
+ ## Built by Ethos Link
405
586
 
406
- Made by the team at [Ethos Link](https://www.ethos-link.com) — practical software for growing businesses. We build tools for hospitality operators who need clear workflows, fast onboarding, and real human support.
587
+ [Ethos Link](https://www.ethos-link.com) builds business software and integrations
588
+ that replace manual handoffs with connected workflows. We help teams automate
589
+ routine work, use their data and improve the processes their business runs on.
407
590
 
408
- We also build [Reviato](https://www.reviato.com), “Capture. Interpret. Act.”.
409
- Turn guest feedback into clear next steps for your team. Collect private appraisals, spot patterns across reviews, and act before small issues turn into public ones.
591
+ - [Reviato](https://www.reviato.com): Read public review trends and collect
592
+ private customer feedback, so your team can decide what to improve.
593
+ - [EthosPlaces](https://www.ethos.place): Give customers the details they need to
594
+ choose and visit your business, from opening hours to directions.
595
+ - [Nuntio](https://www.nuntiohq.com): Draft, review and schedule social posts in
596
+ one workspace, then track each publishing result.