maptain 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 8a0dfd1699f3246011ff0c4b0dd87cff962abd79c81191ca6796fdb1ea4d2bdc
4
+ data.tar.gz: 041486da1610d3e2e37ccbfc22eba6a21dd3266a3230cf4cb6489213ba4e3330
5
+ SHA512:
6
+ metadata.gz: cf8ad48f10150704db1e0e2ff5411c758d434c7e863dcf9c7638d08b1509f3e10a116582135c70bad8c2f31c6c0ac63fc4d30570a637bd2ea9f59e9bdbfdee05
7
+ data.tar.gz: bbef7409decf932bd3083966a49c7307b0517a45ca4b2d5550db96c404dfdd99d367a7a239b3e60d8dd9e853900df58214fac4b2b4fbf7818e39fe888aa5859d
data/CHANGELOG.md ADDED
@@ -0,0 +1,17 @@
1
+ # Changelog
2
+
3
+ ## [Unreleased]
4
+
5
+ ## [0.1.0] — 2026-11-21
6
+
7
+ First release: XML sitemaps for Rails sites in many languages, rebuilt in milliseconds when little has changed.
8
+
9
+ **New**
10
+ - One `Maptain::Source` class per kind of page, whose hreflang links come from the languages of each record, with `config.host_by_locale` for a host per language.
11
+ - Files cut by ranges of ids, so the same records land in the same file at every build, and builds that rewrite only the ranges whose count or latest change moved: 3 changed records out of 30,000 in 24 languages rebuild 430 times faster than with sitemap_generator.
12
+ - Local storage or Active Storage, gzip at the highest level, and a manifest that lets the next build repair one that died midway.
13
+ - IndexNow submissions of the pages that appeared, changed or disappeared, which never fail a build.
14
+ - Checks without a single HTTP request: encoded URLs, W3C dates, hreflang codes, one line per repeated warning.
15
+
16
+ [Unreleased]: https://github.com/levups/maptain/compare/v0.1.0...HEAD
17
+ [0.1.0]: https://github.com/levups/maptain/releases/tag/v0.1.0
data/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License
2
+
3
+ Copyright (c) 2026 Level UP Solutions
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,386 @@
1
+ # <img src="docs/logo.svg" alt="" width="56" align="top"> Maptain
2
+
3
+ Rails sitemaps that rewrite only what changed. Each build asks your database what moved since the last one and
4
+ leaves every other file alone, so it runs hundreds of times faster than regenerating everything. You describe
5
+ each kind of page once, in a small class; Maptain writes one URL per language with its `hreflang` links.
6
+
7
+ ```ruby
8
+ # app/sitemaps/product_sitemap.rb
9
+ class ProductSitemap < Maptain::Source
10
+ scope { Product.published }
11
+ url { |product, locale| product_url(product, locale:) }
12
+ end
13
+ ```
14
+
15
+ ```sh
16
+ bin/rails maptain:build
17
+ ```
18
+
19
+ That writes `public/sitemap.xml` and the files it lists. Run it again an hour later: if nothing changed, it
20
+ takes a few milliseconds and writes nothing.
21
+
22
+ Website, guide and benchmarks: <https://levups.github.io/maptain/>.
23
+
24
+ ## Is it for you?
25
+
26
+ Maptain was written for one site: 30,000 products in 24 languages, 720,000 URLs. It will help most if yours
27
+ looks a bit like that, with records in a database, translated, and changing a little every day.
28
+
29
+ A site of a few hundred pages in one language does not need it: [sitemap_generator](https://github.com/kjvarga/sitemap_generator)
30
+ or a view that lists them will do. Maptain is also young (0.1), so read [what it does not do yet](#not-there-yet)
31
+ before you switch.
32
+
33
+ ## What it does differently
34
+
35
+ - **A page is a record in several languages.** Every language of it lists all the others and itself as
36
+ `hreflang` links, which search engines require before they trust any of them. You never write those links.
37
+ - **Files that stay put.** Each file holds a fixed range of ids, so a record lands in the same file at every
38
+ build, and crawlers only fetch again the files that changed.
39
+ - **Builds that skip what did not move.** One query per source asks the database the count, the sum of the ids
40
+ and the latest change of each range. No callback in your models, so `update_all`, imports and deletions are
41
+ seen too.
42
+ - **Dates that are true.** `lastmod` is the date your content changed. `priority` and `changefreq`, which
43
+ Google ignores, are not written.
44
+ - **IndexNow instead of pings.** Bing, Yandex, Seznam and Naver are told which URLs changed. The ping endpoints
45
+ that older gems call have been closed.
46
+
47
+ Its only runtime dependency is Active Support.
48
+
49
+ ## How long it takes
50
+
51
+ `bench/build.rb` builds the same sitemap with Maptain and with sitemap_generator, on the same machine. When a
52
+ few records changed, Maptain is about 400 times faster: it rewrites 2 files where sitemap_generator writes them
53
+ all again. Once a week, a full build rewrites everything to catch what the database numbers cannot see; that one
54
+ is 15 times faster.
55
+
56
+ On 30,000 records in 24 languages (SQLite in memory, Ruby 4.0.9 without YJIT, an Apple M5 Max, median of 3
57
+ runs; seconds depend on the machine, the ratios hold from one to the next):
58
+
59
+ | Build | Maptain | sitemap_generator | Faster | Files written |
60
+ |---|---|---|---|---|
61
+ | Nothing new | 0.004 s | 150 s | (nothing to write) | 0 |
62
+ | 3 records changed | 0.35 s | 150 s | 430× | 2 |
63
+ | Everything, 720,000 URLs | 9.9 s | 150 s | 15× | 55 (16.5 MiB gzipped) |
64
+
65
+ The URLs there are plain strings. With route helpers and a real database every build takes longer, and the gap
66
+ narrows a little: on a copy of a production app's database, 560,000 URLs took 35 s in full (13 times faster) and 2.8 s after
67
+ three products changed (170 times faster), where sitemap_generator took 468 s. The [benchmarks page](https://levups.github.io/maptain/benchmarks.html) explains
68
+ where the time goes.
69
+
70
+ ## Getting started
71
+
72
+ 1. Add the gem. It needs Ruby 3.3 and Rails 8.0, or newer.
73
+
74
+ ```ruby
75
+ gem "maptain"
76
+ ```
77
+
78
+ 2. Name the host of the site:
79
+
80
+ ```ruby
81
+ # config/initializers/maptain.rb
82
+ Maptain.configure do |config|
83
+ config.host = "https://www.example.com"
84
+ config.locales = %i[en fr de] # default: I18n.available_locales
85
+ config.x_default = :en # the language of hreflang="x-default"; none when left out
86
+ end
87
+ ```
88
+
89
+ 3. Write one class per kind of page in `app/sitemaps`, like `ProductSitemap` above. Its blocks have your route
90
+ helpers.
91
+
92
+ 4. Look at what it would write, without writing it:
93
+
94
+ ```sh
95
+ bin/rails maptain:check
96
+ ```
97
+
98
+ It fails on a URL outside the site, a URL listed twice, a date in the future, and a few other
99
+ [mistakes](#checking).
100
+
101
+ 5. Build, and tell crawlers where the index is, in `public/robots.txt`:
102
+
103
+ ```sh
104
+ bin/rails maptain:build
105
+ ```
106
+
107
+ ```
108
+ Sitemap: https://www.example.com/sitemap.xml
109
+ ```
110
+
111
+ 6. Run it on a schedule. With Solid Queue:
112
+
113
+ ```yaml
114
+ # config/recurring.yml
115
+ production:
116
+ sitemap:
117
+ command: "Maptain.build"
118
+ schedule: every hour
119
+ ```
120
+
121
+ The rest of this README is the guide: languages, dates, files, storage on S3 and the like, IndexNow.
122
+
123
+ ## Coming from sitemap_generator
124
+
125
+ | sitemap_generator | Maptain |
126
+ |---|---|
127
+ | `config/sitemap.rb`, one `add` per URL | one `Maptain::Source` per kind of page, in `app/sitemaps` |
128
+ | `alternates:` written by hand for each URL | `hreflang` links written from `config.locales`, or a source's `locales` |
129
+ | `default_host` | `config.host`, and `config.host_by_locale` for one host per language |
130
+ | `rake sitemap:refresh` | `bin/rails maptain:build`, which rewrites only what changed |
131
+ | `lastmod:` | `updated_at` by default, or `lastmod :column` |
132
+ | `priority:`, `changefreq:` | left out: Google ignores them |
133
+ | pinging search engines | IndexNow, with `config.index_now_key` |
134
+ | `sitemaps_host`, adapters | `config.sitemaps_url`, and any Active Storage service |
135
+
136
+ sitemap_generator names its index `sitemap.xml.gz`, Maptain `sitemap.xml`: change the `Sitemap:` line of
137
+ `robots.txt`, and delete the old files once the first build has written the new ones.
138
+
139
+ ## Sources
140
+
141
+ A source is a class in `app/sitemaps`. Its blocks run in an object that has your route helpers, already set to
142
+ `config.host`, or in the `url` block to the host of the language being written. That block also runs with its
143
+ language as `I18n.locale`, so a slug read from a translated attribute comes out in the language of its page; a
144
+ language that is not in `I18n.available_locales` runs in the app's.
145
+
146
+ ```ruby
147
+ class ProductSitemap < Maptain::Source
148
+ # The records: an Active Record relation. Add the `includes` your other blocks need.
149
+ scope { Product.published.includes(:photos) }
150
+
151
+ # The URL of a record in a language. A path gets the host of its language; nil leaves that language out.
152
+ url { |product, locale| product_url(product, locale:) }
153
+
154
+ # The languages a record exists in. Without it, every configured language.
155
+ locales { |product| product.translated_locales }
156
+
157
+ # The column that holds the date the content changed. Default: updated_at.
158
+ lastmod :content_updated_at
159
+
160
+ # Image URLs, for Google Images.
161
+ images { |product| product.photos.map(&:url) }
162
+ end
163
+ ```
164
+
165
+ The model needs an integer primary key: files are cut by ranges of ids.
166
+
167
+ ### Pages that are not records
168
+
169
+ The scope can be any list. Such a source is one file, rendered at each build and published when it differs:
170
+
171
+ ```ruby
172
+ class PageSitemap < Maptain::Source
173
+ scope { %i[root about contact] }
174
+ url { |page, locale| public_send(:"#{page}_url", locale:) }
175
+ end
176
+ ```
177
+
178
+ ## Languages
179
+
180
+ Every record gets one URL per language of `config.locales`, or of its source's `locales` block. Each `<url>`
181
+ then lists every language of its page, itself included:
182
+
183
+ ```xml
184
+ <url>
185
+ <loc>https://www.example.com/fr/products/42</loc>
186
+ <lastmod>2026-11-03T09:12:44Z</lastmod>
187
+ <xhtml:link rel="alternate" hreflang="en" href="https://www.example.com/en/products/42"/>
188
+ <xhtml:link rel="alternate" hreflang="fr" href="https://www.example.com/fr/products/42"/>
189
+ <xhtml:link rel="alternate" hreflang="x-default" href="https://www.example.com/en/products/42"/>
190
+ </url>
191
+ ```
192
+
193
+ A page in one language gets no `hreflang` link. A page that does not exist in the `x_default` language gets
194
+ no `x-default` link. A Rails locale such as `pt_BR` is written `hreflang="pt-BR"`, and two languages that share
195
+ a URL list it once.
196
+
197
+ A URL your block writes by hand is percent-encoded: spaces, accents and control characters cannot break a file.
198
+
199
+ ### One host per language
200
+
201
+ When some languages are served from a host of their own, name those hosts:
202
+
203
+ ```ruby
204
+ config.host = "https://www.example.com"
205
+ config.host_by_locale = {fr: "https://fr.example.com", de: "https://de.example.com"}
206
+ ```
207
+
208
+ The other languages stay on `config.host`, and so does the index. The `robots.txt` of every host names that one
209
+ index: a sitemap may list the pages of a host only when that host's `robots.txt` points to it. Each key is one
210
+ of `config.locales`: a build refuses any other.
211
+
212
+ ## The date of a page
213
+
214
+ `lastmod` is read from `updated_at`. Name another column when the content has its own: `lastmod
215
+ :content_updated_at`.
216
+
217
+ A column that moves when the content does not, such as an `updated_at` touched by a view counter, makes every
218
+ build rewrite files for nothing and tells search engines a date that is not true. Give such a model a column of
219
+ its own for content changes.
220
+
221
+ ### When the date depends on more than one column
222
+
223
+ `lastmod` also takes a block. The build then needs to know what to watch in the database, as an SQL expression:
224
+
225
+ ```ruby
226
+ class ProductSitemap < Maptain::Source
227
+ scope { Product.published }
228
+ url { |product, locale| product_url(product, locale:) }
229
+ lastmod { |product| [product.updated_at, product.reviewed_at].compact.max }
230
+ fingerprint "GREATEST(products.updated_at, COALESCE(products.reviewed_at, products.updated_at))"
231
+ end
232
+ ```
233
+
234
+ ## Files
235
+
236
+ A source named `ProductSitemap` writes `sitemap-product-00000.xml.gz`, `sitemap-product-00001.xml.gz` and so
237
+ on, one per range of ids; `slug "products"` changes the middle part, `config.filename` the first. The size of a
238
+ range comes from the number of languages, so that a file stays under 50,000 URLs and 50 MiB: 50,000 ids with one
239
+ language, 554 with 24. `shard_size 1_000` sets it yourself. A range that still outgrows a file continues in
240
+ `sitemap-product-00000-2.xml.gz`.
241
+
242
+ `sitemap.xml` is the index. It lists every file with the latest `lastmod` it holds, so a crawler fetches again
243
+ only the files that changed.
244
+
245
+ ## Building
246
+
247
+ ```sh
248
+ bin/rails maptain:build # rewrite the files whose records changed
249
+ bin/rails maptain:rebuild # rewrite everything
250
+ ```
251
+
252
+ `Maptain.build` does the same from Ruby and returns what it did (`full`, `written`, `kept`, `deleted`,
253
+ `url_count`, `submitted`, `warnings`).
254
+
255
+ ### How a build decides
256
+
257
+ For each source, one query groups the records by range of ids and returns the count, the sum of the ids and the
258
+ latest change of each. A range whose answer is the one the last build stored keeps its file. The others are
259
+ rendered again, and a file that comes out the same is not uploaded (a full build uploads every file). In the
260
+ index, a file that changed takes the latest `lastmod` it holds, or the time of the build when a record left it,
261
+ so that crawlers read it again.
262
+
263
+ Some changes leave those numbers alone: `update_columns`, a change in another table that the `url`, `locales` or
264
+ `images` blocks read, a new `url` block, a file someone removed from the storage. Name the other table in
265
+ `fingerprint`, or `touch` the record from it. For the rest, a full build comes once a week by itself:
266
+
267
+ ```ruby
268
+ config.rebuild_every = 1.day # default: a week; nil leaves it to maptain:rebuild
269
+ ```
270
+
271
+ After deploying a change to a `url` block, run `maptain:rebuild` rather than wait.
272
+
273
+ A build notes the files it is about to add in `sitemap-manifest.json`, its memory of what each file held, then
274
+ writes the sitemap files, then the index, then removes the files no range fills any more, then writes the
275
+ manifest again. A build that fails midway is repaired by the next one. Changing a host, the languages, the
276
+ size of a range or the version of Maptain rewrites everything it moves once.
277
+
278
+ ## Storage
279
+
280
+ Files go to `public/` by default.
281
+
282
+ ### Through Active Storage
283
+
284
+ An app that already uses Active Storage can send the sitemaps through one of its services, with the credentials
285
+ it already has. Give them a service of their own in `config/storage.yml`, in the bucket you want:
286
+
287
+ ```yaml
288
+ sitemaps:
289
+ service: S3
290
+ bucket: example-sitemaps
291
+ region: eu-west-3
292
+ public: true
293
+ access_key_id: <%= Rails.application.credentials.dig(:aws, :access_key_id) %>
294
+ secret_access_key: <%= Rails.application.credentials.dig(:aws, :secret_access_key) %>
295
+ ```
296
+
297
+ ```ruby
298
+ Maptain.configure do |config|
299
+ config.storage = Maptain::Storage::ActiveStorage.new(:sitemaps, prefix: "sitemaps")
300
+ config.sitemaps_url = "https://cdn.example.com/sitemaps" # where those files are served from
301
+ end
302
+ ```
303
+
304
+ No blob is created: the files keep their names as keys, and nothing is written to the database. Any service
305
+ works that serves a key at a stable URL: S3, Google Cloud Storage, Azure, a mirror of them. The Disk service
306
+ does not, it scatters keys in subfolders: keep the default for files on disk.
307
+
308
+ The `Sitemap:` line of `robots.txt` then names the index there: a sitemap on another host, or in a folder of the
309
+ site, may list the pages of the site only when its `robots.txt` points to it.
310
+
311
+ A storage is any object with `read(name)`, `write(name, path)` and `delete(name)`.
312
+
313
+ ## IndexNow
314
+
315
+ [IndexNow](https://www.indexnow.org) tells Bing, Yandex, Seznam and Naver that a URL changed. Google does not
316
+ take part. Choose a key of 8 to 128 characters among a-z, A-Z, 0-9 and -:
317
+
318
+ ```ruby
319
+ config.index_now_key = "c1f0e2b4a6d84f3e9b7a5c3d1e0f2a4b"
320
+ ```
321
+
322
+ Each build then publishes the key as `<key>.txt` at the root of the site, and sends, from the files it rewrote,
323
+ the URLs whose `lastmod` is later than the previous build, and those that appeared or went away whatever their
324
+ `lastmod`. The first build sends nothing, nor does a new configuration beyond the pages whose `lastmod` moved,
325
+ and only URLs on the hosts of the site are sent.
326
+
327
+ When the sitemaps are served from elsewhere (`config.sitemaps_url`), the key cannot follow them: IndexNow reads
328
+ it on the site. Put `<key>.txt` in `public/` and set `config.index_now_key_location` to its path,
329
+ `"/<key>.txt"`; a build warns until you do.
330
+
331
+ With one host per language, every host must serve the key: IndexNow reads it on the host of the URLs it is given.
332
+
333
+ A refusal or an endpoint that cannot be reached is a warning, never a failed build: the URLs it did not take are
334
+ kept in the manifest and sent again with the next build. Once the endpoint cannot be reached, a build asks it
335
+ nothing more.
336
+
337
+ ## Checking
338
+
339
+ ```sh
340
+ bin/rails maptain:check
341
+ ```
342
+
343
+ It renders every sitemap without publishing or submitting anything, and fails on:
344
+
345
+ - a URL outside the hosts of the site, listed twice in a source, or longer than 2,048 characters, and a page whose
346
+ languages all share one URL;
347
+ - an `hreflang` that is not a language code (`en`, `pt-BR`, `zh-Hant`, `es-419`);
348
+ - a `lastmod` in the future, or one that is not a W3C date, which is then left out;
349
+ - an index with more than the 50,000 files it may hold.
350
+
351
+ When many URLs of a source are outside the hosts of the site, listed twice, too long or dated in the future, one
352
+ line names the first and counts the others. A build reports the same, for the files it rewrites, in `warnings`.
353
+
354
+ Maptain never requests a page of the site. Whether a URL answers 200 or carries a `noindex` is what Search
355
+ Console reports, from the visits crawlers make anyway.
356
+
357
+ ## Instrumentation
358
+
359
+ `build.maptain` wraps a build, with `full` and `result` in its payload; `shard.maptain` wraps each file
360
+ range it renders, with `source` and `shard`.
361
+
362
+ ```ruby
363
+ ActiveSupport::Notifications.subscribe("build.maptain") do |event|
364
+ built = event.payload[:result] # nil when the build raised
365
+ Rails.logger.info "Sitemap: #{built.written.size} files written in #{event.duration.round} ms" if built
366
+ end
367
+ ```
368
+
369
+ ## Not there yet
370
+
371
+ Video and news sitemaps, models without an integer primary key, and building several ranges in parallel.
372
+
373
+ ## Development
374
+
375
+ ```sh
376
+ bundle exec rake test
377
+ bundle exec rubocop
378
+ bundle exec ruby bench/build.rb # also builds the same sitemap with sitemap_generator: a few minutes
379
+ bundle exec rake docs:highlight # after editing a code block in docs/
380
+ ```
381
+
382
+ ## License
383
+
384
+ MIT. See [LICENSE](LICENSE).
385
+
386
+ Made by [Level UP Solutions](https://github.com/levups).