indexmap 0.8.2 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +85 -0
- data/README.md +281 -94
- data/UPGRADE.md +361 -0
- data/docs/architecture.md +55 -0
- data/docs/deployment.md +223 -0
- data/docs/domain-terms.md +27 -0
- data/docs/dynamic-sitemaps.md +302 -0
- data/docs/search-engine-ping.md +368 -0
- data/docs/storage.md +94 -0
- data/lib/generators/indexmap/install_generator.rb +1 -1
- data/lib/generators/indexmap/templates/initializer.rb.tt +18 -18
- data/lib/indexmap/artifact.rb +9 -0
- data/lib/indexmap/configuration.rb +12 -1
- data/lib/indexmap/creator.rb +45 -42
- data/lib/indexmap/index_now_configuration.rb +3 -1
- data/lib/indexmap/output.rb +19 -5
- data/lib/indexmap/parser.rb +112 -53
- data/lib/indexmap/pinger/base.rb +6 -5
- data/lib/indexmap/pinger/google.rb +12 -8
- data/lib/indexmap/pinger/index_now.rb +97 -52
- data/lib/indexmap/publication.rb +89 -0
- data/lib/indexmap/rails.rb +6 -0
- data/lib/indexmap/railtie.rb +7 -0
- data/lib/indexmap/robots.rb +67 -0
- data/lib/indexmap/robots_controller.rb +13 -0
- data/lib/indexmap/sitemaps_controller.rb +57 -0
- data/lib/indexmap/storage/active_storage.rb +132 -1
- data/lib/indexmap/storage/filesystem.rb +182 -11
- data/lib/indexmap/storage/memory.rb +1 -1
- data/lib/indexmap/task_runner.rb +21 -28
- data/lib/indexmap/validator/duplicates.rb +83 -0
- data/lib/indexmap/validator.rb +171 -59
- data/lib/indexmap/version.rb +1 -1
- data/lib/indexmap/writer.rb +89 -34
- data/lib/indexmap.rb +14 -1
- data/lib/tasks/indexmap_tasks.rake +21 -2
- data/test/indexmap/correctness_test.rb +153 -0
- data/test/indexmap/install_generator_test.rb +36 -4
- data/test/indexmap/parser_test.rb +77 -0
- data/test/indexmap/pinger/google_test.rb +22 -0
- data/test/indexmap/pinger/index_now_test.rb +168 -0
- data/test/indexmap/publication_test.rb +280 -0
- data/test/indexmap/rails_test.rb +22 -0
- data/test/indexmap/robots_test.rb +73 -0
- data/test/indexmap/sitemaps_controller_test.rb +50 -0
- data/test/indexmap/storage_test.rb +27 -0
- data/test/indexmap/task_runner_test.rb +24 -0
- data/test/indexmap/validator_test.rb +14 -0
- data/test/indexmap/writer_test.rb +42 -0
- data/test/integration/queue_checks.rb +244 -0
- metadata +33 -51
- data/lib/indexmap/storage/file.rb +0 -11
- data/test/release_task_test.rb +0 -86
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 0a88a19e175e22156b0b054041e4d673365eee9089e5fc42347a31f1583dc2e9
|
|
4
|
+
data.tar.gz: 1404999d7723a994cdf5763715dc61c6366a435dace94d9f0796bd82aad69753
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: e610924028d149f000c15f5d0c570232457040aa01b77769c271fe94d4ecc9da4ce067a78167b9c659b892c75f923e6d5b0861838ed0ff7563d7c63753c24673
|
|
7
|
+
data.tar.gz: 5c4986f19bcd67274948b9c4c9a6db4c8a1b7855f3b3c20774179fdc13ee0c300df4c10bf79249d42f358faf4636ea0ba60788eb692fe055a5a987814c6ddfc6
|
data/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,91 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [0.10.0] - 2026-11-04
|
|
9
|
+
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- require Ruby 4 and adopt Gemkit
|
|
14
|
+
|
|
15
|
+
- require Ruby 4 and adopt Gemkit ([#27](https://github.com/ethos-link/indexmap/pull/27))
|
|
16
|
+
|
|
17
|
+
|
|
18
|
+
|
|
19
|
+
|
|
20
|
+
### Fixed
|
|
21
|
+
|
|
22
|
+
- push the version tag during preparation
|
|
23
|
+
|
|
24
|
+
- push the version tag during preparation ([#26](https://github.com/ethos-link/indexmap/pull/26))
|
|
25
|
+
|
|
26
|
+
|
|
27
|
+
|
|
28
|
+
## [0.9.0] - 2026-11-04
|
|
29
|
+
|
|
30
|
+
|
|
31
|
+
### Added
|
|
32
|
+
|
|
33
|
+
- validate and expose public crawler output
|
|
34
|
+
|
|
35
|
+
- validate and expose public crawler output ([#21](https://github.com/ethos-link/indexmap/pull/21))
|
|
36
|
+
|
|
37
|
+
- stream and atomically publish dynamic inventories
|
|
38
|
+
|
|
39
|
+
- stream and atomically publish dynamic inventories ([#23](https://github.com/ethos-link/indexmap/pull/23))
|
|
40
|
+
|
|
41
|
+
|
|
42
|
+
|
|
43
|
+
|
|
44
|
+
### Changed
|
|
45
|
+
|
|
46
|
+
- reconcile writer branch with merged sitemap fixes
|
|
47
|
+
|
|
48
|
+
- synchronize publication branch with resolved writer base
|
|
49
|
+
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
|
|
53
|
+
### Documentation
|
|
54
|
+
|
|
55
|
+
- explain background sitemap generation ([#19](https://github.com/ethos-link/indexmap/pull/19))
|
|
56
|
+
|
|
57
|
+
- explain public key setup in Rails ([#20](https://github.com/ethos-link/indexmap/pull/20))
|
|
58
|
+
|
|
59
|
+
- clarify usage limits and implementation status
|
|
60
|
+
|
|
61
|
+
- clarify retries are disabled by default
|
|
62
|
+
|
|
63
|
+
- document unreleased APIs and sitemap deployment choices
|
|
64
|
+
|
|
65
|
+
|
|
66
|
+
|
|
67
|
+
|
|
68
|
+
### Fixed
|
|
69
|
+
|
|
70
|
+
- consume sitemap section entries once
|
|
71
|
+
|
|
72
|
+
- consume sitemap section entries once ([#22](https://github.com/ethos-link/indexmap/pull/22))
|
|
73
|
+
|
|
74
|
+
- separate review preparation from release tagging
|
|
75
|
+
|
|
76
|
+
- prepare and push releases from main
|
|
77
|
+
|
|
78
|
+
- install git-cliff before running release tests
|
|
79
|
+
|
|
80
|
+
- prepare and push releases from main ([#25](https://github.com/ethos-link/indexmap/pull/25))
|
|
81
|
+
|
|
82
|
+
|
|
83
|
+
|
|
84
|
+
|
|
85
|
+
### fix
|
|
86
|
+
|
|
87
|
+
- harden publication and Rails task loading
|
|
88
|
+
|
|
89
|
+
- harden sitemap publication and Rails integration ([#24](https://github.com/ethos-link/indexmap/pull/24))
|
|
90
|
+
|
|
91
|
+
|
|
92
|
+
|
|
8
93
|
## [0.8.2] - 2026-07-22
|
|
9
94
|
|
|
10
95
|
|
data/README.md
CHANGED
|
@@ -5,7 +5,7 @@
|
|
|
5
5
|
|
|
6
6
|
`indexmap` is a small Ruby gem for generating XML sitemaps from explicit Ruby data.
|
|
7
7
|
|
|
8
|
-
|
|
8
|
+
Use it from Ruby or an optional Rails 8 integration for:
|
|
9
9
|
|
|
10
10
|
- deterministic sitemap output
|
|
11
11
|
- plain Ruby configuration
|
|
@@ -14,6 +14,9 @@ It is designed for Rails apps that want:
|
|
|
14
14
|
|
|
15
15
|
By default, `indexmap` writes a sitemap index plus one or more child sitemap files. For simpler sites, it also supports `:single_file` mode, which writes a single `urlset` directly to `sitemap.xml`.
|
|
16
16
|
|
|
17
|
+
See [CHANGELOG.md](CHANGELOG.md) for published changes and [UPGRADE.md](UPGRADE.md)
|
|
18
|
+
for host migration steps.
|
|
19
|
+
|
|
17
20
|
## Installation
|
|
18
21
|
|
|
19
22
|
Add this line to your application's Gemfile:
|
|
@@ -44,6 +47,14 @@ directory prefix such as `sitemaps/`.
|
|
|
44
47
|
|
|
45
48
|
## Documentation
|
|
46
49
|
|
|
50
|
+
- [How Indexmap works](docs/architecture.md): pipeline diagram and entrypoints.
|
|
51
|
+
- [Domain terms](docs/domain-terms.md): sections, partitions, outputs, publications,
|
|
52
|
+
roots, and entry types.
|
|
53
|
+
- [Deployment guide](docs/deployment.md): development and production, static and
|
|
54
|
+
database-backed sources, static files or Rails delivery, and robots.txt.
|
|
55
|
+
- [Storage contract](docs/storage.md): legacy, streaming, atomic, and pruning APIs.
|
|
56
|
+
- [Dynamic sitemaps](docs/dynamic-sitemaps.md): streaming and partitions, atomic
|
|
57
|
+
publication, Rails delivery, retention, jobs, and host adoption.
|
|
47
58
|
- [Search engine ping](docs/search-engine-ping.md): Google Search Console
|
|
48
59
|
credentials, Google property configuration, and IndexNow key setup.
|
|
49
60
|
- [UPGRADE.md](UPGRADE.md): host-app migration steps for public contract
|
|
@@ -64,10 +75,15 @@ sections = [
|
|
|
64
75
|
)
|
|
65
76
|
]
|
|
66
77
|
|
|
67
|
-
Indexmap
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
78
|
+
Indexmap.configure do |config|
|
|
79
|
+
config.base_url = "https://example.com"
|
|
80
|
+
config.storage = Indexmap::Storage::Filesystem.new(
|
|
81
|
+
path: "public", public_url: config.base_url
|
|
82
|
+
)
|
|
83
|
+
config.sections = sections
|
|
84
|
+
end
|
|
85
|
+
|
|
86
|
+
Indexmap.create
|
|
71
87
|
```
|
|
72
88
|
|
|
73
89
|
## Rails Usage
|
|
@@ -79,7 +95,7 @@ bin/rails generate indexmap:install
|
|
|
79
95
|
```
|
|
80
96
|
|
|
81
97
|
The generator creates `config/initializers/indexmap.rb` with an idempotent
|
|
82
|
-
`
|
|
98
|
+
`Indexmap::HostConfiguration.apply` loader and adds conditional `indexmap/tasks`
|
|
83
99
|
loading to the application's `Rakefile`. This preserves `bin/rails --tasks` and
|
|
84
100
|
the `indexmap:*` tasks without loading Indexmap during normal Rails boot.
|
|
85
101
|
|
|
@@ -88,12 +104,10 @@ Customize the `Indexmap.configure` block inside the generated initializer:
|
|
|
88
104
|
```ruby
|
|
89
105
|
Indexmap.configure do |config|
|
|
90
106
|
config.base_url = -> { "https://example.com" }
|
|
91
|
-
config.storage =
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
)
|
|
96
|
-
end
|
|
107
|
+
config.storage = Indexmap::Storage::Filesystem.new(
|
|
108
|
+
path: Rails.public_path,
|
|
109
|
+
public_url: config.base_url
|
|
110
|
+
)
|
|
97
111
|
config.sections = -> do
|
|
98
112
|
[
|
|
99
113
|
Indexmap::Section.new(
|
|
@@ -111,7 +125,7 @@ Runtime callers and jobs must apply the generated configuration before using
|
|
|
111
125
|
Indexmap:
|
|
112
126
|
|
|
113
127
|
```ruby
|
|
114
|
-
|
|
128
|
+
Indexmap::HostConfiguration.apply
|
|
115
129
|
Indexmap.create
|
|
116
130
|
```
|
|
117
131
|
|
|
@@ -122,14 +136,188 @@ Then run:
|
|
|
122
136
|
|
|
123
137
|
```bash
|
|
124
138
|
bin/rails indexmap:sitemap:create
|
|
125
|
-
bin/rails indexmap:sitemap:format
|
|
126
139
|
bin/rails indexmap:sitemap:validate
|
|
127
140
|
```
|
|
128
141
|
|
|
129
|
-
`indexmap:sitemap:
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
142
|
+
Creation already validates. `indexmap:sitemap:format` optionally pretty-prints
|
|
143
|
+
legacy XML after publication; it rejects atomic outputs.
|
|
144
|
+
|
|
145
|
+
`indexmap:sitemap:create` is the main task. It stages partitioned XML on temporary
|
|
146
|
+
disk, validates the complete result, then publishes to configured storage.
|
|
147
|
+
Generation and validation failures leave published files untouched. Legacy
|
|
148
|
+
publication remains sequential; enable atomic mode for immutable children and
|
|
149
|
+
revision-checked root activation. See [dynamic sitemaps](docs/dynamic-sitemaps.md).
|
|
150
|
+
|
|
151
|
+
### Background Generation
|
|
152
|
+
|
|
153
|
+
Keep the refresh job and its scheduling in the host application. Indexmap does
|
|
154
|
+
not select a queue adapter, run workers, or deduplicate jobs. Apply the generated
|
|
155
|
+
configuration inside the job. Use `Indexmap::TaskRunner.new.create` for the
|
|
156
|
+
rake task's complete lifecycle, including `after_create` callbacks and IndexNow
|
|
157
|
+
key-file handling; `Indexmap.create` only generates the selected output.
|
|
158
|
+
|
|
159
|
+
#### GoodJob
|
|
160
|
+
|
|
161
|
+
With GoodJob 4.22.1 or newer, use its public labelled concurrency rules:
|
|
162
|
+
|
|
163
|
+
```ruby
|
|
164
|
+
class SitemapRefreshJob < ApplicationJob
|
|
165
|
+
include GoodJob::ActiveJobExtensions::Concurrency
|
|
166
|
+
|
|
167
|
+
good_job_concurrency_rule(
|
|
168
|
+
perform_limit: 1,
|
|
169
|
+
label: "sitemap-refresh",
|
|
170
|
+
apply_label: true
|
|
171
|
+
)
|
|
172
|
+
|
|
173
|
+
good_job_concurrency_rule(
|
|
174
|
+
total_limit: 1,
|
|
175
|
+
label: -> { "sitemap-refresh:#{arguments.first}" },
|
|
176
|
+
apply_label: true
|
|
177
|
+
)
|
|
178
|
+
|
|
179
|
+
def perform(_version)
|
|
180
|
+
Indexmap::HostConfiguration.apply
|
|
181
|
+
Indexmap::TaskRunner.new.create
|
|
182
|
+
end
|
|
183
|
+
end
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
Enqueue with a release identifier: `SitemapRefreshJob.perform_later(version)`.
|
|
187
|
+
The version label limits unfinished work for that release; the shared label
|
|
188
|
+
limits generation across releases to one execution at a time. A newer release
|
|
189
|
+
can enqueue while an older refresh is unfinished. A repeat enqueue is allowed
|
|
190
|
+
after completion: these controls do not provide permanent once-per-release
|
|
191
|
+
tracking. Use `perform_later`; direct `perform_now` calls bypass GoodJob's limits.
|
|
192
|
+
See [GoodJob concurrency controls](https://github.com/bensheldon/good_job#concurrency-controls).
|
|
193
|
+
|
|
194
|
+
#### Solid Queue
|
|
195
|
+
|
|
196
|
+
With Solid Queue 1.7 or newer, serialize writes to the shared sitemap output:
|
|
197
|
+
|
|
198
|
+
```ruby
|
|
199
|
+
class SitemapRefreshJob < ApplicationJob
|
|
200
|
+
limits_concurrency to: 1, key: "sitemap-refresh",
|
|
201
|
+
duration: 30.minutes, on_conflict: :block
|
|
202
|
+
|
|
203
|
+
def perform(_version)
|
|
204
|
+
Indexmap::HostConfiguration.apply
|
|
205
|
+
Indexmap::TaskRunner.new.create
|
|
206
|
+
end
|
|
207
|
+
end
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
Choose a duration longer than the expected generation time. `:block` retains
|
|
211
|
+
every refresh and runs them serially; it does not suppress duplicate enqueues.
|
|
212
|
+
Using `on_conflict: :discard` coalesces conflicting requests, including requests
|
|
213
|
+
for newer releases when the key is shared. A version-based key such as
|
|
214
|
+
`->(version) { "sitemap-refresh:#{version}" }` scopes conflicts to that release,
|
|
215
|
+
but permits different releases to write concurrently. Select that tradeoff
|
|
216
|
+
explicitly; it is not equivalent to GoodJob's two rules above. Direct
|
|
217
|
+
`perform_now` calls bypass queue coordination.
|
|
218
|
+
See [Solid Queue concurrency controls](https://github.com/rails/solid_queue#concurrency-controls).
|
|
219
|
+
|
|
220
|
+
#### Triggering Refreshes
|
|
221
|
+
|
|
222
|
+
Enqueue from one release task, a recurring schedule, or application startup.
|
|
223
|
+
Rails' `after_initialize` hook can enqueue from a worker application without
|
|
224
|
+
depending on a backend's worker callbacks:
|
|
225
|
+
|
|
226
|
+
```ruby
|
|
227
|
+
Rails.application.config.after_initialize do
|
|
228
|
+
if Rails.env.production? && ENV["PROCESS_ROLE"] == "worker"
|
|
229
|
+
SitemapRefreshJob.perform_later(ENV.fetch("RELEASE_VERSION")) do |job|
|
|
230
|
+
raise job.enqueue_error if job.enqueue_error
|
|
231
|
+
end
|
|
232
|
+
end
|
|
233
|
+
end
|
|
234
|
+
```
|
|
235
|
+
|
|
236
|
+
`PROCESS_ROLE` and `RELEASE_VERSION` are application-defined environment values.
|
|
237
|
+
Configure them in the deployment, with the worker role flag limited to worker
|
|
238
|
+
processes. `after_initialize` runs for every Rails process, including commands;
|
|
239
|
+
it indicates application initialization, not worker readiness. Complete database
|
|
240
|
+
preparation before starting workers. Suppressed enqueues are expected; report
|
|
241
|
+
actual enqueue errors. Avoid scheduling the same refresh from both startup and
|
|
242
|
+
a release task.
|
|
243
|
+
|
|
244
|
+
Use storage shared by all serving containers, such as object storage through
|
|
245
|
+
the Active Storage adapter. A job container's local filesystem is not shared
|
|
246
|
+
with web containers unless the deployment mounts the same persistent storage.
|
|
247
|
+
|
|
248
|
+
#### Operating Practices And Current Limits
|
|
249
|
+
|
|
250
|
+
Generate database-backed content after database preparation, through one
|
|
251
|
+
host-owned refresh job. Load the existing configuration inside that job and
|
|
252
|
+
query eligible records when it runs. Pass release or output identifiers rather
|
|
253
|
+
than relations, URL inventories, or credentials. Keep queue selection, recurring
|
|
254
|
+
schedules, and any application authorization context in the host.
|
|
255
|
+
|
|
256
|
+
Use a shared concurrency key or label for jobs writing the same destination,
|
|
257
|
+
across releases. Prefer blocking or rescheduling conflicts to discarding a newer
|
|
258
|
+
refresh. Backend controls do not protect manual rake invocations, and Solid
|
|
259
|
+
Queue's concurrency duration can expire before an unusually long job finishes.
|
|
260
|
+
Avoid concurrent writers through all entrypoints in legacy mode: an interrupted
|
|
261
|
+
sequential publication can leave mixed generations. Opt-in atomic mode checks
|
|
262
|
+
root revisions under a storage lock and retains previous generations. See the
|
|
263
|
+
[publication and retention contract](docs/dynamic-sitemaps.md#atomic-publication-and-storage).
|
|
264
|
+
|
|
265
|
+
Keep generation and search-engine submission separate. A refresh retry should
|
|
266
|
+
not automatically resend successful IndexNow batches. Make `after_create`
|
|
267
|
+
callbacks safe to repeat: they run after publication, and a callback failure does
|
|
268
|
+
not roll published files back. Validate the deployed sitemap and preview IndexNow
|
|
269
|
+
selection before submitting. See [selection and attempt reports](docs/search-engine-ping.md#inspect-selection-and-actual-attempts).
|
|
270
|
+
|
|
271
|
+
Batch source queries with native Rails APIs and select the fields needed for
|
|
272
|
+
URLs and dates. Supply meaningful content dates; do not use the refresh time as
|
|
273
|
+
`lastmod`. Each sitemap must stay within 50,000 entries and 52,428,800 uncompressed
|
|
274
|
+
bytes. Index mode partitions automatically; single-file mode rejects an output
|
|
275
|
+
that needs splitting. Measure the complete refresh before adding parallel jobs or
|
|
276
|
+
caches. Supply lazy sources to benefit from incremental generation and validation;
|
|
277
|
+
app-owned arrays and explicit inventory-returning APIs still retain their data.
|
|
278
|
+
|
|
279
|
+
Generation uses the same synchronous entrypoints with GoodJob and Solid Queue;
|
|
280
|
+
the examples above describe their different concurrency behavior. Verify the
|
|
281
|
+
host's actual adapter, workers, shared storage, failure reporting, and intended
|
|
282
|
+
system context in integration tests. Gem tests alone do not establish those
|
|
283
|
+
deployment guarantees.
|
|
284
|
+
|
|
285
|
+
### Dynamic Robots
|
|
286
|
+
|
|
287
|
+
Indexmap can serve `/robots.txt` from configuration without querying content or
|
|
288
|
+
reading sitemap storage. Crawling defaults to disabled. Inside the generated
|
|
289
|
+
`Indexmap.configure` block, configure the site's policy:
|
|
290
|
+
|
|
291
|
+
```ruby
|
|
292
|
+
config.robots.indexable = -> { Rails.env.production? && ENV["PUBLIC_INDEXING"] == "1" }
|
|
293
|
+
config.robots.groups = [
|
|
294
|
+
{user_agent: "*", allow: ["/"], disallow: ["/admin/", "/search"]}
|
|
295
|
+
]
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
The sitemap URL defaults to `base_url` plus `index_filename`. Override it with
|
|
299
|
+
`config.robots.sitemaps = ["https://example.com/sitemap.xml"]` when needed.
|
|
300
|
+
Groups and sitemap lists also accept callables. Keep them configuration-only;
|
|
301
|
+
do not query models in these callables.
|
|
302
|
+
|
|
303
|
+
In the application's `config/routes.rb`, load and mount the endpoint explicitly:
|
|
304
|
+
|
|
305
|
+
```ruby
|
|
306
|
+
require "indexmap/robots_controller"
|
|
307
|
+
Indexmap::HostConfiguration.apply
|
|
308
|
+
|
|
309
|
+
Rails.application.routes.draw do
|
|
310
|
+
get "/robots.txt", to: Indexmap::RobotsController.action(:show)
|
|
311
|
+
end
|
|
312
|
+
```
|
|
313
|
+
|
|
314
|
+
If the configuration loader lives outside an initializer, require that file
|
|
315
|
+
before applying it. Remove `public/robots.txt` so it cannot shadow the route.
|
|
316
|
+
The controller supports GET, HEAD, and conditional requests, uses a one-hour
|
|
317
|
+
public cache, and does not inherit application authentication or browser gates.
|
|
318
|
+
Set `PUBLIC_INDEXING` only for the intended public deployment; production-mode
|
|
319
|
+
previews should remain disabled. Robots rules do not replace authorization or
|
|
320
|
+
page-level indexing directives.
|
|
133
321
|
|
|
134
322
|
### Default Index Mode
|
|
135
323
|
|
|
@@ -138,6 +326,24 @@ This is the default behavior. `indexmap` writes:
|
|
|
138
326
|
- `sitemap.xml` as a sitemap index
|
|
139
327
|
- one or more child sitemap files from `config.sections`
|
|
140
328
|
|
|
329
|
+
Section entries can be enumerables. Indexmap consumes each section once and
|
|
330
|
+
calculates its index `lastmod` in that same pass. For database sources, return a
|
|
331
|
+
fresh enumerable from the configured sections callable on every generation:
|
|
332
|
+
|
|
333
|
+
```ruby
|
|
334
|
+
config.sections = -> do
|
|
335
|
+
entries = PublicPage.indexable.find_each.lazy.map do |page|
|
|
336
|
+
Indexmap::Entry.new(loc: page.canonical_url, lastmod: page.updated_at)
|
|
337
|
+
end
|
|
338
|
+
[Indexmap::Section.new(filename: "sitemap-pages.xml", entries: entries)]
|
|
339
|
+
end
|
|
340
|
+
```
|
|
341
|
+
|
|
342
|
+
`PublicPage`, its eligibility scope, and canonical URL method are host-app
|
|
343
|
+
examples. Keep queries and routing decisions in the host. Generation stages
|
|
344
|
+
bounded partitions and validates incrementally, using temporary disk for global
|
|
345
|
+
duplicate checks. See [memory and disk limits](docs/dynamic-sitemaps.md#configure-generation).
|
|
346
|
+
|
|
141
347
|
### Single-File Mode
|
|
142
348
|
|
|
143
349
|
For sites that only want one `sitemap.xml` file:
|
|
@@ -145,7 +351,7 @@ For sites that only want one `sitemap.xml` file:
|
|
|
145
351
|
```ruby
|
|
146
352
|
Indexmap.configure do |config|
|
|
147
353
|
config.base_url = -> { "https://example.com" }
|
|
148
|
-
config.storage =
|
|
354
|
+
config.storage = Indexmap::Storage::Filesystem.new(path: Rails.public_path, public_url: config.base_url)
|
|
149
355
|
config.format = :single_file
|
|
150
356
|
config.entries = -> do
|
|
151
357
|
[
|
|
@@ -168,7 +374,7 @@ outputs write through the same configured storage as the default output.
|
|
|
168
374
|
```ruby
|
|
169
375
|
Indexmap.configure do |config|
|
|
170
376
|
config.base_url = -> { "https://example.com" }
|
|
171
|
-
config.storage =
|
|
377
|
+
config.storage = Indexmap::Storage::Filesystem.new(path: Rails.root.join("storage/sitemaps"), public_url: config.base_url)
|
|
172
378
|
config.sections = -> { Sitemap.sections }
|
|
173
379
|
|
|
174
380
|
config.output :insights_data do |output|
|
|
@@ -194,8 +400,10 @@ Indexmap.create(:insights_data)
|
|
|
194
400
|
Named outputs inherit `base_url` and `format` from the main configuration unless
|
|
195
401
|
you override them. Storage is configured once and shared by every output.
|
|
196
402
|
|
|
197
|
-
`Indexmap.create`
|
|
198
|
-
|
|
403
|
+
`Indexmap.create` stages XML, validates it, then publishes the selected output.
|
|
404
|
+
Only `Indexmap::TaskRunner#create` (also used by the create task) handles the
|
|
405
|
+
IndexNow key file and `after_create` callbacks. Formatting is a separate optional
|
|
406
|
+
operation for legacy outputs; atomic outputs must be regenerated.
|
|
199
407
|
|
|
200
408
|
### Storage
|
|
201
409
|
|
|
@@ -243,17 +451,26 @@ storage.delete(filename)
|
|
|
243
451
|
storage.public_url(filename)
|
|
244
452
|
```
|
|
245
453
|
|
|
454
|
+
Atomic publication, pruning, and streamed HTTP delivery require additional
|
|
455
|
+
capabilities. See the [storage contract](docs/storage.md) for signatures,
|
|
456
|
+
return values, and adapter requirements. Each operation instance resolves a
|
|
457
|
+
storage callable once and shares the adapter with its collaborators. Create a
|
|
458
|
+
fresh runner/pinger after changing destinations or tenant context. Direct reads
|
|
459
|
+
of `config.storage` still resolve the callable; construct an adapter once in the
|
|
460
|
+
host loader when its settings are fixed for the process.
|
|
461
|
+
|
|
246
462
|
### Deferred Dynamic Sections
|
|
247
463
|
|
|
248
464
|
Use `after_create` when `indexmap:sitemap:create` should publish the default
|
|
249
465
|
sitemap first, then schedule slower dynamic sections for the background. The
|
|
250
|
-
callback runs only after the
|
|
251
|
-
|
|
466
|
+
callback runs only after the default output has been validated and published
|
|
467
|
+
and configured IndexNow key-file handling has succeeded. It also runs for
|
|
468
|
+
`Indexmap::TaskRunner#create`; `Indexmap.create` does not run callbacks.
|
|
252
469
|
|
|
253
470
|
```ruby
|
|
254
471
|
Indexmap.configure do |config|
|
|
255
472
|
config.base_url = -> { "https://example.com" }
|
|
256
|
-
config.storage =
|
|
473
|
+
config.storage = Indexmap::Storage::Filesystem.new(path: Rails.root.join("storage/sitemaps"), public_url: config.base_url)
|
|
257
474
|
config.sections = -> { Sitemap.sections }
|
|
258
475
|
|
|
259
476
|
config.output :insights_data do |output|
|
|
@@ -273,13 +490,17 @@ Then the job can stay small:
|
|
|
273
490
|
```ruby
|
|
274
491
|
class Insights::SitemapRefreshJob < ApplicationJob
|
|
275
492
|
def perform
|
|
493
|
+
Indexmap::HostConfiguration.apply
|
|
276
494
|
Indexmap.create(:insights_data)
|
|
277
495
|
end
|
|
278
496
|
end
|
|
279
497
|
```
|
|
280
498
|
|
|
281
|
-
|
|
282
|
-
|
|
499
|
+
The default output and named output are independent roots. List both public
|
|
500
|
+
root URLs in `config.robots.sitemaps` (or the static robots file) and submit each
|
|
501
|
+
separately if needed; named outputs are not automatically linked into the default
|
|
502
|
+
index. Publish the named output before advertising its URL. A deferred job can
|
|
503
|
+
fail after the default root is published, so monitor it separately.
|
|
283
504
|
|
|
284
505
|
## Validation And Parsing
|
|
285
506
|
|
|
@@ -312,6 +533,11 @@ The built-in validator checks for:
|
|
|
312
533
|
See [Search engine ping](docs/search-engine-ping.md) for Google credential
|
|
313
534
|
setup, Search Console property configuration, and IndexNow key provisioning.
|
|
314
535
|
|
|
536
|
+
For Rails apps with sitemaps outside `public/`, see
|
|
537
|
+
[serving an IndexNow key from `public/`](docs/search-engine-ping.md#rails-public-key-file-with-separate-sitemap-storage).
|
|
538
|
+
Key discovery searches the configured sitemap storage, so an existing public
|
|
539
|
+
key file needs explicit configuration in that setup.
|
|
540
|
+
|
|
315
541
|
Available rake tasks:
|
|
316
542
|
|
|
317
543
|
```bash
|
|
@@ -326,84 +552,45 @@ If Google credentials are blank, `indexmap:google:ping` skips Google
|
|
|
326
552
|
submission. If neither a configured IndexNow key nor a valid key file is
|
|
327
553
|
present, `indexmap:index_now:ping` skips IndexNow submission.
|
|
328
554
|
|
|
329
|
-
##
|
|
330
|
-
|
|
331
|
-
Run tests:
|
|
332
|
-
|
|
333
|
-
```bash
|
|
334
|
-
bundle exec rake test
|
|
335
|
-
```
|
|
336
|
-
|
|
337
|
-
Run lint:
|
|
338
|
-
|
|
339
|
-
```bash
|
|
340
|
-
bundle exec rake standard
|
|
341
|
-
```
|
|
342
|
-
|
|
343
|
-
Run the full default task:
|
|
344
|
-
|
|
345
|
-
```bash
|
|
346
|
-
bundle exec rake
|
|
347
|
-
```
|
|
348
|
-
|
|
349
|
-
Tests generate a coverage report automatically.
|
|
350
|
-
|
|
351
|
-
Note: `Gemfile.lock` is intentionally not tracked for this gem, following normal Ruby library conventions.
|
|
555
|
+
## License
|
|
352
556
|
|
|
353
|
-
|
|
557
|
+
MIT License, see [LICENSE.txt](LICENSE.txt)
|
|
354
558
|
|
|
355
|
-
|
|
356
|
-
[commitlint](https://github.com/arandilopez/commitlint) gem to enforce
|
|
357
|
-
Conventional Commits on every commit. We also use
|
|
358
|
-
[Standard Ruby](https://standardrb.com/) to keep code style consistent. CI
|
|
359
|
-
validates commit messages, Standard Ruby, tests, and git-cliff changelog
|
|
360
|
-
generation on pull requests and pushes to main/master.
|
|
559
|
+
## Development and releases
|
|
361
560
|
|
|
362
|
-
|
|
561
|
+
Shared tooling uses published [Ethos Gemkit](https://github.com/ethos-link/ethos-gemkit).
|
|
562
|
+
Keep the exact Gemfile version and workflow commit pins on the same release.
|
|
363
563
|
|
|
364
|
-
```
|
|
564
|
+
```sh
|
|
565
|
+
bundle install
|
|
365
566
|
bundle exec lefthook install
|
|
567
|
+
bundle exec rake verify
|
|
366
568
|
```
|
|
367
569
|
|
|
368
|
-
|
|
369
|
-
|
|
370
|
-
Releases are tag-driven and published by GitHub Actions to RubyGems.
|
|
371
|
-
Local release commands never publish directly.
|
|
372
|
-
|
|
373
|
-
Install [git-cliff](https://git-cliff.org/) locally before preparing a
|
|
374
|
-
release. The release task prepends the next `CHANGELOG.md` section from
|
|
375
|
-
Conventional Commits.
|
|
570
|
+
The PostgreSQL integration suite covers Active Storage, GoodJob and Solid Queue:
|
|
376
571
|
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
Then run one of:
|
|
382
|
-
|
|
383
|
-
```bash
|
|
384
|
-
bundle exec rake 'release:prepare[patch]'
|
|
385
|
-
bundle exec rake 'release:prepare[minor]'
|
|
386
|
-
bundle exec rake 'release:prepare[major]'
|
|
387
|
-
bundle exec rake 'release:prepare[0.1.0]'
|
|
572
|
+
```sh
|
|
573
|
+
BUNDLE_GEMFILE=gemfiles/integration.gemfile bundle install
|
|
574
|
+
INDEXMAP_DATABASE_URL=postgresql://postgres@localhost:5432/indexmap_test \
|
|
575
|
+
BUNDLE_GEMFILE=gemfiles/integration.gemfile bundle exec ruby script/integration
|
|
388
576
|
```
|
|
389
577
|
|
|
390
|
-
|
|
578
|
+
Use a disposable `indexmap_test` database; the integration suite recreates it.
|
|
391
579
|
|
|
392
|
-
|
|
393
|
-
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
397
|
-
The `Release` workflow then runs tests, publishes the gem to RubyGems,
|
|
398
|
-
and creates the GitHub release from the changelog entry.
|
|
399
|
-
|
|
400
|
-
## License
|
|
401
|
-
|
|
402
|
-
MIT License, see [LICENSE.txt](LICENSE.txt)
|
|
580
|
+
From clean, current `main`, run `bundle exec rake 'release:prepare[patch]'`
|
|
581
|
+
(or `minor`, `major`, or an explicit version). Review and merge the generated
|
|
582
|
+
release PR to publish through Actions. Shared prerequisites, registry credentials
|
|
583
|
+
and failed-job recovery are in the [Gemkit release guide](https://github.com/ethos-link/ethos-gemkit#releasing-a-gem).
|
|
403
584
|
|
|
404
|
-
##
|
|
585
|
+
## Built by Ethos Link
|
|
405
586
|
|
|
406
|
-
|
|
587
|
+
[Ethos Link](https://www.ethos-link.com) builds business software and integrations
|
|
588
|
+
that replace manual handoffs with connected workflows. We help teams automate
|
|
589
|
+
routine work, use their data and improve the processes their business runs on.
|
|
407
590
|
|
|
408
|
-
|
|
409
|
-
|
|
591
|
+
- [Reviato](https://www.reviato.com): Read public review trends and collect
|
|
592
|
+
private customer feedback, so your team can decide what to improve.
|
|
593
|
+
- [EthosPlaces](https://www.ethos.place): Give customers the details they need to
|
|
594
|
+
choose and visit your business, from opening hours to directions.
|
|
595
|
+
- [Nuntio](https://www.nuntiohq.com): Draft, review and schedule social posts in
|
|
596
|
+
one workspace, then track each publishing result.
|