maptain 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +17 -0
- data/LICENSE +21 -0
- data/README.md +386 -0
- data/lib/maptain/builder.rb +357 -0
- data/lib/maptain/configuration.rb +100 -0
- data/lib/maptain/errors.rb +9 -0
- data/lib/maptain/index_now.rb +45 -0
- data/lib/maptain/manifest.rb +58 -0
- data/lib/maptain/page.rb +11 -0
- data/lib/maptain/railtie.rb +17 -0
- data/lib/maptain/shard_writer.rb +68 -0
- data/lib/maptain/source.rb +250 -0
- data/lib/maptain/storage/active_storage.rb +42 -0
- data/lib/maptain/storage/local.rb +33 -0
- data/lib/maptain/storage/null.rb +16 -0
- data/lib/maptain/tasks.rake +26 -0
- data/lib/maptain/version.rb +5 -0
- data/lib/maptain/xml.rb +102 -0
- data/lib/maptain.rb +56 -0
- metadata +80 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 8a0dfd1699f3246011ff0c4b0dd87cff962abd79c81191ca6796fdb1ea4d2bdc
|
|
4
|
+
data.tar.gz: 041486da1610d3e2e37ccbfc22eba6a21dd3266a3230cf4cb6489213ba4e3330
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: cf8ad48f10150704db1e0e2ff5411c758d434c7e863dcf9c7638d08b1509f3e10a116582135c70bad8c2f31c6c0ac63fc4d30570a637bd2ea9f59e9bdbfdee05
|
|
7
|
+
data.tar.gz: bbef7409decf932bd3083966a49c7307b0517a45ca4b2d5550db96c404dfdd99d367a7a239b3e60d8dd9e853900df58214fac4b2b4fbf7818e39fe888aa5859d
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## [Unreleased]
|
|
4
|
+
|
|
5
|
+
## [0.1.0] — 2026-11-21
|
|
6
|
+
|
|
7
|
+
First release: XML sitemaps for Rails sites in many languages, rebuilt in milliseconds when little has changed.
|
|
8
|
+
|
|
9
|
+
**New**
|
|
10
|
+
- One `Maptain::Source` class per kind of page, whose hreflang links come from the languages of each record, with `config.host_by_locale` for a host per language.
|
|
11
|
+
- Files cut by ranges of ids, so the same records land in the same file at every build, and builds that rewrite only the ranges whose count or latest change moved: 3 changed records out of 30,000 in 24 languages rebuild 430 times faster than with sitemap_generator.
|
|
12
|
+
- Local storage or Active Storage, gzip at the highest level, and a manifest that lets the next build repair one that died midway.
|
|
13
|
+
- IndexNow submissions of the pages that appeared, changed or disappeared, which never fail a build.
|
|
14
|
+
- Checks without a single HTTP request: encoded URLs, W3C dates, hreflang codes, one line per repeated warning.
|
|
15
|
+
|
|
16
|
+
[Unreleased]: https://github.com/levups/maptain/compare/v0.1.0...HEAD
|
|
17
|
+
[0.1.0]: https://github.com/levups/maptain/releases/tag/v0.1.0
|
data/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
The MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Level UP Solutions
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
|
13
|
+
all copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
|
21
|
+
THE SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,386 @@
|
|
|
1
|
+
# <img src="docs/logo.svg" alt="" width="56" align="top"> Maptain
|
|
2
|
+
|
|
3
|
+
Rails sitemaps that rewrite only what changed. Each build asks your database what moved since the last one and
|
|
4
|
+
leaves every other file alone, so it runs hundreds of times faster than regenerating everything. You describe
|
|
5
|
+
each kind of page once, in a small class; Maptain writes one URL per language with its `hreflang` links.
|
|
6
|
+
|
|
7
|
+
```ruby
|
|
8
|
+
# app/sitemaps/product_sitemap.rb
|
|
9
|
+
class ProductSitemap < Maptain::Source
|
|
10
|
+
scope { Product.published }
|
|
11
|
+
url { |product, locale| product_url(product, locale:) }
|
|
12
|
+
end
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
```sh
|
|
16
|
+
bin/rails maptain:build
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
That writes `public/sitemap.xml` and the files it lists. Run it again an hour later: if nothing changed, it
|
|
20
|
+
takes a few milliseconds and writes nothing.
|
|
21
|
+
|
|
22
|
+
Website, guide and benchmarks: <https://levups.github.io/maptain/>.
|
|
23
|
+
|
|
24
|
+
## Is it for you?
|
|
25
|
+
|
|
26
|
+
Maptain was written for one site: 30,000 products in 24 languages, 720,000 URLs. It will help most if yours
|
|
27
|
+
looks a bit like that, with records in a database, translated, and changing a little every day.
|
|
28
|
+
|
|
29
|
+
A site of a few hundred pages in one language does not need it: [sitemap_generator](https://github.com/kjvarga/sitemap_generator)
|
|
30
|
+
or a view that lists them will do. Maptain is also young (0.1), so read [what it does not do yet](#not-there-yet)
|
|
31
|
+
before you switch.
|
|
32
|
+
|
|
33
|
+
## What it does differently
|
|
34
|
+
|
|
35
|
+
- **A page is a record in several languages.** Every language of it lists all the others and itself as
|
|
36
|
+
`hreflang` links, which search engines require before they trust any of them. You never write those links.
|
|
37
|
+
- **Files that stay put.** Each file holds a fixed range of ids, so a record lands in the same file at every
|
|
38
|
+
build, and crawlers only fetch again the files that changed.
|
|
39
|
+
- **Builds that skip what did not move.** One query per source asks the database the count, the sum of the ids
|
|
40
|
+
and the latest change of each range. No callback in your models, so `update_all`, imports and deletions are
|
|
41
|
+
seen too.
|
|
42
|
+
- **Dates that are true.** `lastmod` is the date your content changed. `priority` and `changefreq`, which
|
|
43
|
+
Google ignores, are not written.
|
|
44
|
+
- **IndexNow instead of pings.** Bing, Yandex, Seznam and Naver are told which URLs changed. The ping endpoints
|
|
45
|
+
that older gems call have been closed.
|
|
46
|
+
|
|
47
|
+
Its only runtime dependency is Active Support.
|
|
48
|
+
|
|
49
|
+
## How long it takes
|
|
50
|
+
|
|
51
|
+
`bench/build.rb` builds the same sitemap with Maptain and with sitemap_generator, on the same machine. When a
|
|
52
|
+
few records changed, Maptain is about 400 times faster: it rewrites 2 files where sitemap_generator writes them
|
|
53
|
+
all again. Once a week, a full build rewrites everything to catch what the database numbers cannot see; that one
|
|
54
|
+
is 15 times faster.
|
|
55
|
+
|
|
56
|
+
On 30,000 records in 24 languages (SQLite in memory, Ruby 4.0.9 without YJIT, an Apple M5 Max, median of 3
|
|
57
|
+
runs; seconds depend on the machine, the ratios hold from one to the next):
|
|
58
|
+
|
|
59
|
+
| Build | Maptain | sitemap_generator | Faster | Files written |
|
|
60
|
+
|---|---|---|---|---|
|
|
61
|
+
| Nothing new | 0.004 s | 150 s | (nothing to write) | 0 |
|
|
62
|
+
| 3 records changed | 0.35 s | 150 s | 430× | 2 |
|
|
63
|
+
| Everything, 720,000 URLs | 9.9 s | 150 s | 15× | 55 (16.5 MiB gzipped) |
|
|
64
|
+
|
|
65
|
+
The URLs there are plain strings. With route helpers and a real database every build takes longer, and the gap
|
|
66
|
+
narrows a little: on a copy of a production app's database, 560,000 URLs took 35 s in full (13 times faster) and 2.8 s after
|
|
67
|
+
three products changed (170 times faster), where sitemap_generator took 468 s. The [benchmarks page](https://levups.github.io/maptain/benchmarks.html) explains
|
|
68
|
+
where the time goes.
|
|
69
|
+
|
|
70
|
+
## Getting started
|
|
71
|
+
|
|
72
|
+
1. Add the gem. It needs Ruby 3.3 and Rails 8.0, or newer.
|
|
73
|
+
|
|
74
|
+
```ruby
|
|
75
|
+
gem "maptain"
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
2. Name the host of the site:
|
|
79
|
+
|
|
80
|
+
```ruby
|
|
81
|
+
# config/initializers/maptain.rb
|
|
82
|
+
Maptain.configure do |config|
|
|
83
|
+
config.host = "https://www.example.com"
|
|
84
|
+
config.locales = %i[en fr de] # default: I18n.available_locales
|
|
85
|
+
config.x_default = :en # the language of hreflang="x-default"; none when left out
|
|
86
|
+
end
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
3. Write one class per kind of page in `app/sitemaps`, like `ProductSitemap` above. Its blocks have your route
|
|
90
|
+
helpers.
|
|
91
|
+
|
|
92
|
+
4. Look at what it would write, without writing it:
|
|
93
|
+
|
|
94
|
+
```sh
|
|
95
|
+
bin/rails maptain:check
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
It fails on a URL outside the site, a URL listed twice, a date in the future, and a few other
|
|
99
|
+
[mistakes](#checking).
|
|
100
|
+
|
|
101
|
+
5. Build, and tell crawlers where the index is, in `public/robots.txt`:
|
|
102
|
+
|
|
103
|
+
```sh
|
|
104
|
+
bin/rails maptain:build
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
```
|
|
108
|
+
Sitemap: https://www.example.com/sitemap.xml
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
6. Run it on a schedule. With Solid Queue:
|
|
112
|
+
|
|
113
|
+
```yaml
|
|
114
|
+
# config/recurring.yml
|
|
115
|
+
production:
|
|
116
|
+
sitemap:
|
|
117
|
+
command: "Maptain.build"
|
|
118
|
+
schedule: every hour
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
The rest of this README is the guide: languages, dates, files, storage on S3 and the like, IndexNow.
|
|
122
|
+
|
|
123
|
+
## Coming from sitemap_generator
|
|
124
|
+
|
|
125
|
+
| sitemap_generator | Maptain |
|
|
126
|
+
|---|---|
|
|
127
|
+
| `config/sitemap.rb`, one `add` per URL | one `Maptain::Source` per kind of page, in `app/sitemaps` |
|
|
128
|
+
| `alternates:` written by hand for each URL | `hreflang` links written from `config.locales`, or a source's `locales` |
|
|
129
|
+
| `default_host` | `config.host`, and `config.host_by_locale` for one host per language |
|
|
130
|
+
| `rake sitemap:refresh` | `bin/rails maptain:build`, which rewrites only what changed |
|
|
131
|
+
| `lastmod:` | `updated_at` by default, or `lastmod :column` |
|
|
132
|
+
| `priority:`, `changefreq:` | left out: Google ignores them |
|
|
133
|
+
| pinging search engines | IndexNow, with `config.index_now_key` |
|
|
134
|
+
| `sitemaps_host`, adapters | `config.sitemaps_url`, and any Active Storage service |
|
|
135
|
+
|
|
136
|
+
sitemap_generator names its index `sitemap.xml.gz`, Maptain `sitemap.xml`: change the `Sitemap:` line of
|
|
137
|
+
`robots.txt`, and delete the old files once the first build has written the new ones.
|
|
138
|
+
|
|
139
|
+
## Sources
|
|
140
|
+
|
|
141
|
+
A source is a class in `app/sitemaps`. Its blocks run in an object that has your route helpers, already set to
|
|
142
|
+
`config.host`, or in the `url` block to the host of the language being written. That block also runs with its
|
|
143
|
+
language as `I18n.locale`, so a slug read from a translated attribute comes out in the language of its page; a
|
|
144
|
+
language that is not in `I18n.available_locales` runs in the app's.
|
|
145
|
+
|
|
146
|
+
```ruby
|
|
147
|
+
class ProductSitemap < Maptain::Source
|
|
148
|
+
# The records: an Active Record relation. Add the `includes` your other blocks need.
|
|
149
|
+
scope { Product.published.includes(:photos) }
|
|
150
|
+
|
|
151
|
+
# The URL of a record in a language. A path gets the host of its language; nil leaves that language out.
|
|
152
|
+
url { |product, locale| product_url(product, locale:) }
|
|
153
|
+
|
|
154
|
+
# The languages a record exists in. Without it, every configured language.
|
|
155
|
+
locales { |product| product.translated_locales }
|
|
156
|
+
|
|
157
|
+
# The column that holds the date the content changed. Default: updated_at.
|
|
158
|
+
lastmod :content_updated_at
|
|
159
|
+
|
|
160
|
+
# Image URLs, for Google Images.
|
|
161
|
+
images { |product| product.photos.map(&:url) }
|
|
162
|
+
end
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
The model needs an integer primary key: files are cut by ranges of ids.
|
|
166
|
+
|
|
167
|
+
### Pages that are not records
|
|
168
|
+
|
|
169
|
+
The scope can be any list. Such a source is one file, rendered at each build and published when it differs:
|
|
170
|
+
|
|
171
|
+
```ruby
|
|
172
|
+
class PageSitemap < Maptain::Source
|
|
173
|
+
scope { %i[root about contact] }
|
|
174
|
+
url { |page, locale| public_send(:"#{page}_url", locale:) }
|
|
175
|
+
end
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
## Languages
|
|
179
|
+
|
|
180
|
+
Every record gets one URL per language of `config.locales`, or of its source's `locales` block. Each `<url>`
|
|
181
|
+
then lists every language of its page, itself included:
|
|
182
|
+
|
|
183
|
+
```xml
|
|
184
|
+
<url>
|
|
185
|
+
<loc>https://www.example.com/fr/products/42</loc>
|
|
186
|
+
<lastmod>2026-11-03T09:12:44Z</lastmod>
|
|
187
|
+
<xhtml:link rel="alternate" hreflang="en" href="https://www.example.com/en/products/42"/>
|
|
188
|
+
<xhtml:link rel="alternate" hreflang="fr" href="https://www.example.com/fr/products/42"/>
|
|
189
|
+
<xhtml:link rel="alternate" hreflang="x-default" href="https://www.example.com/en/products/42"/>
|
|
190
|
+
</url>
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
A page in one language gets no `hreflang` link. A page that does not exist in the `x_default` language gets
|
|
194
|
+
no `x-default` link. A Rails locale such as `pt_BR` is written `hreflang="pt-BR"`, and two languages that share
|
|
195
|
+
a URL list it once.
|
|
196
|
+
|
|
197
|
+
A URL your block writes by hand is percent-encoded: spaces, accents and control characters cannot break a file.
|
|
198
|
+
|
|
199
|
+
### One host per language
|
|
200
|
+
|
|
201
|
+
When some languages are served from a host of their own, name those hosts:
|
|
202
|
+
|
|
203
|
+
```ruby
|
|
204
|
+
config.host = "https://www.example.com"
|
|
205
|
+
config.host_by_locale = {fr: "https://fr.example.com", de: "https://de.example.com"}
|
|
206
|
+
```
|
|
207
|
+
|
|
208
|
+
The other languages stay on `config.host`, and so does the index. The `robots.txt` of every host names that one
|
|
209
|
+
index: a sitemap may list the pages of a host only when that host's `robots.txt` points to it. Each key is one
|
|
210
|
+
of `config.locales`: a build refuses any other.
|
|
211
|
+
|
|
212
|
+
## The date of a page
|
|
213
|
+
|
|
214
|
+
`lastmod` is read from `updated_at`. Name another column when the content has its own: `lastmod
|
|
215
|
+
:content_updated_at`.
|
|
216
|
+
|
|
217
|
+
A column that moves when the content does not, such as an `updated_at` touched by a view counter, makes every
|
|
218
|
+
build rewrite files for nothing and tells search engines a date that is not true. Give such a model a column of
|
|
219
|
+
its own for content changes.
|
|
220
|
+
|
|
221
|
+
### When the date depends on more than one column
|
|
222
|
+
|
|
223
|
+
`lastmod` also takes a block. The build then needs to know what to watch in the database, as an SQL expression:
|
|
224
|
+
|
|
225
|
+
```ruby
|
|
226
|
+
class ProductSitemap < Maptain::Source
|
|
227
|
+
scope { Product.published }
|
|
228
|
+
url { |product, locale| product_url(product, locale:) }
|
|
229
|
+
lastmod { |product| [product.updated_at, product.reviewed_at].compact.max }
|
|
230
|
+
fingerprint "GREATEST(products.updated_at, COALESCE(products.reviewed_at, products.updated_at))"
|
|
231
|
+
end
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
## Files
|
|
235
|
+
|
|
236
|
+
A source named `ProductSitemap` writes `sitemap-product-00000.xml.gz`, `sitemap-product-00001.xml.gz` and so
|
|
237
|
+
on, one per range of ids; `slug "products"` changes the middle part, `config.filename` the first. The size of a
|
|
238
|
+
range comes from the number of languages, so that a file stays under 50,000 URLs and 50 MiB: 50,000 ids with one
|
|
239
|
+
language, 554 with 24. `shard_size 1_000` sets it yourself. A range that still outgrows a file continues in
|
|
240
|
+
`sitemap-product-00000-2.xml.gz`.
|
|
241
|
+
|
|
242
|
+
`sitemap.xml` is the index. It lists every file with the latest `lastmod` it holds, so a crawler fetches again
|
|
243
|
+
only the files that changed.
|
|
244
|
+
|
|
245
|
+
## Building
|
|
246
|
+
|
|
247
|
+
```sh
|
|
248
|
+
bin/rails maptain:build # rewrite the files whose records changed
|
|
249
|
+
bin/rails maptain:rebuild # rewrite everything
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
`Maptain.build` does the same from Ruby and returns what it did (`full`, `written`, `kept`, `deleted`,
|
|
253
|
+
`url_count`, `submitted`, `warnings`).
|
|
254
|
+
|
|
255
|
+
### How a build decides
|
|
256
|
+
|
|
257
|
+
For each source, one query groups the records by range of ids and returns the count, the sum of the ids and the
|
|
258
|
+
latest change of each. A range whose answer is the one the last build stored keeps its file. The others are
|
|
259
|
+
rendered again, and a file that comes out the same is not uploaded (a full build uploads every file). In the
|
|
260
|
+
index, a file that changed takes the latest `lastmod` it holds, or the time of the build when a record left it,
|
|
261
|
+
so that crawlers read it again.
|
|
262
|
+
|
|
263
|
+
Some changes leave those numbers alone: `update_columns`, a change in another table that the `url`, `locales` or
|
|
264
|
+
`images` blocks read, a new `url` block, a file someone removed from the storage. Name the other table in
|
|
265
|
+
`fingerprint`, or `touch` the record from it. For the rest, a full build comes once a week by itself:
|
|
266
|
+
|
|
267
|
+
```ruby
|
|
268
|
+
config.rebuild_every = 1.day # default: a week; nil leaves it to maptain:rebuild
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
After deploying a change to a `url` block, run `maptain:rebuild` rather than wait.
|
|
272
|
+
|
|
273
|
+
A build notes the files it is about to add in `sitemap-manifest.json`, its memory of what each file held, then
|
|
274
|
+
writes the sitemap files, then the index, then removes the files no range fills any more, then writes the
|
|
275
|
+
manifest again. A build that fails midway is repaired by the next one. Changing a host, the languages, the
|
|
276
|
+
size of a range or the version of Maptain rewrites everything it moves once.
|
|
277
|
+
|
|
278
|
+
## Storage
|
|
279
|
+
|
|
280
|
+
Files go to `public/` by default.
|
|
281
|
+
|
|
282
|
+
### Through Active Storage
|
|
283
|
+
|
|
284
|
+
An app that already uses Active Storage can send the sitemaps through one of its services, with the credentials
|
|
285
|
+
it already has. Give them a service of their own in `config/storage.yml`, in the bucket you want:
|
|
286
|
+
|
|
287
|
+
```yaml
|
|
288
|
+
sitemaps:
|
|
289
|
+
service: S3
|
|
290
|
+
bucket: example-sitemaps
|
|
291
|
+
region: eu-west-3
|
|
292
|
+
public: true
|
|
293
|
+
access_key_id: <%= Rails.application.credentials.dig(:aws, :access_key_id) %>
|
|
294
|
+
secret_access_key: <%= Rails.application.credentials.dig(:aws, :secret_access_key) %>
|
|
295
|
+
```
|
|
296
|
+
|
|
297
|
+
```ruby
|
|
298
|
+
Maptain.configure do |config|
|
|
299
|
+
config.storage = Maptain::Storage::ActiveStorage.new(:sitemaps, prefix: "sitemaps")
|
|
300
|
+
config.sitemaps_url = "https://cdn.example.com/sitemaps" # where those files are served from
|
|
301
|
+
end
|
|
302
|
+
```
|
|
303
|
+
|
|
304
|
+
No blob is created: the files keep their names as keys, and nothing is written to the database. Any service
|
|
305
|
+
works that serves a key at a stable URL: S3, Google Cloud Storage, Azure, a mirror of them. The Disk service
|
|
306
|
+
does not, it scatters keys in subfolders: keep the default for files on disk.
|
|
307
|
+
|
|
308
|
+
The `Sitemap:` line of `robots.txt` then names the index there: a sitemap on another host, or in a folder of the
|
|
309
|
+
site, may list the pages of the site only when its `robots.txt` points to it.
|
|
310
|
+
|
|
311
|
+
A storage is any object with `read(name)`, `write(name, path)` and `delete(name)`.
|
|
312
|
+
|
|
313
|
+
## IndexNow
|
|
314
|
+
|
|
315
|
+
[IndexNow](https://www.indexnow.org) tells Bing, Yandex, Seznam and Naver that a URL changed. Google does not
|
|
316
|
+
take part. Choose a key of 8 to 128 characters among a-z, A-Z, 0-9 and -:
|
|
317
|
+
|
|
318
|
+
```ruby
|
|
319
|
+
config.index_now_key = "c1f0e2b4a6d84f3e9b7a5c3d1e0f2a4b"
|
|
320
|
+
```
|
|
321
|
+
|
|
322
|
+
Each build then publishes the key as `<key>.txt` at the root of the site, and sends, from the files it rewrote,
|
|
323
|
+
the URLs whose `lastmod` is later than the previous build, and those that appeared or went away whatever their
|
|
324
|
+
`lastmod`. The first build sends nothing, nor does a new configuration beyond the pages whose `lastmod` moved,
|
|
325
|
+
and only URLs on the hosts of the site are sent.
|
|
326
|
+
|
|
327
|
+
When the sitemaps are served from elsewhere (`config.sitemaps_url`), the key cannot follow them: IndexNow reads
|
|
328
|
+
it on the site. Put `<key>.txt` in `public/` and set `config.index_now_key_location` to its path,
|
|
329
|
+
`"/<key>.txt"`; a build warns until you do.
|
|
330
|
+
|
|
331
|
+
With one host per language, every host must serve the key: IndexNow reads it on the host of the URLs it is given.
|
|
332
|
+
|
|
333
|
+
A refusal or an endpoint that cannot be reached is a warning, never a failed build: the URLs it did not take are
|
|
334
|
+
kept in the manifest and sent again with the next build. Once the endpoint cannot be reached, a build asks it
|
|
335
|
+
nothing more.
|
|
336
|
+
|
|
337
|
+
## Checking
|
|
338
|
+
|
|
339
|
+
```sh
|
|
340
|
+
bin/rails maptain:check
|
|
341
|
+
```
|
|
342
|
+
|
|
343
|
+
It renders every sitemap without publishing or submitting anything, and fails on:
|
|
344
|
+
|
|
345
|
+
- a URL outside the hosts of the site, listed twice in a source, or longer than 2,048 characters, and a page whose
|
|
346
|
+
languages all share one URL;
|
|
347
|
+
- an `hreflang` that is not a language code (`en`, `pt-BR`, `zh-Hant`, `es-419`);
|
|
348
|
+
- a `lastmod` in the future, or one that is not a W3C date, which is then left out;
|
|
349
|
+
- an index with more than the 50,000 files it may hold.
|
|
350
|
+
|
|
351
|
+
When many URLs of a source are outside the hosts of the site, listed twice, too long or dated in the future, one
|
|
352
|
+
line names the first and counts the others. A build reports the same, for the files it rewrites, in `warnings`.
|
|
353
|
+
|
|
354
|
+
Maptain never requests a page of the site. Whether a URL answers 200 or carries a `noindex` is what Search
|
|
355
|
+
Console reports, from the visits crawlers make anyway.
|
|
356
|
+
|
|
357
|
+
## Instrumentation
|
|
358
|
+
|
|
359
|
+
`build.maptain` wraps a build, with `full` and `result` in its payload; `shard.maptain` wraps each file
|
|
360
|
+
range it renders, with `source` and `shard`.
|
|
361
|
+
|
|
362
|
+
```ruby
|
|
363
|
+
ActiveSupport::Notifications.subscribe("build.maptain") do |event|
|
|
364
|
+
built = event.payload[:result] # nil when the build raised
|
|
365
|
+
Rails.logger.info "Sitemap: #{built.written.size} files written in #{event.duration.round} ms" if built
|
|
366
|
+
end
|
|
367
|
+
```
|
|
368
|
+
|
|
369
|
+
## Not there yet
|
|
370
|
+
|
|
371
|
+
Video and news sitemaps, models without an integer primary key, and building several ranges in parallel.
|
|
372
|
+
|
|
373
|
+
## Development
|
|
374
|
+
|
|
375
|
+
```sh
|
|
376
|
+
bundle exec rake test
|
|
377
|
+
bundle exec rubocop
|
|
378
|
+
bundle exec ruby bench/build.rb # also builds the same sitemap with sitemap_generator: a few minutes
|
|
379
|
+
bundle exec rake docs:highlight # after editing a code block in docs/
|
|
380
|
+
```
|
|
381
|
+
|
|
382
|
+
## License
|
|
383
|
+
|
|
384
|
+
MIT. See [LICENSE](LICENSE).
|
|
385
|
+
|
|
386
|
+
Made by [Level UP Solutions](https://github.com/levups).
|