indexmap 0.3.1 → 0.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +3 -5
- data/README.md +71 -20
- data/lib/indexmap/index_now_configuration.rb +1 -1
- data/lib/indexmap/pinger/google.rb +34 -3
- data/lib/indexmap/pinger/index_now.rb +78 -15
- data/lib/indexmap/task_runner.rb +17 -7
- data/lib/indexmap/version.rb +1 -1
- data/lib/tasks/indexmap_tasks.rake +99 -9
- data/test/indexmap/pinger/google_test.rb +48 -2
- data/test/indexmap/pinger/index_now_test.rb +120 -2
- data/test/indexmap/task_runner_test.rb +16 -1
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 513c19d20a8373690c66fa99a0251db934298589a19139c2b77654340ce7006a
|
|
4
|
+
data.tar.gz: '087bd725e6ecaf81ff07510f78f7c5fa1d613730cd7b735c9f0ed60ecb3f4676'
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 2f4065163991559ca5a1df38f1f01aafbdc539936f40aa81d78a3c5aa4abe34bdbc7cdb0df25d970a8a243e1b552c1946ec97c640f8d64350a46ef0d8de36006
|
|
7
|
+
data.tar.gz: d0ebff2036cefb30689a22ebddf313c89ac45a6dda11277a4a5322b6c39c6257e6d05e3615e0167ca876f985b14dc5ac21eba096f947876ab72be32bc6a73c33
|
data/CHANGELOG.md
CHANGED
|
@@ -5,14 +5,12 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
-
## [0.
|
|
8
|
+
## [0.4.1] - 2026-04-22
|
|
9
9
|
|
|
10
10
|
|
|
11
|
-
###
|
|
11
|
+
### Added
|
|
12
12
|
|
|
13
|
-
-
|
|
14
|
-
|
|
15
|
-
- harden indexmap runtime defaults and test coverage (#3)
|
|
13
|
+
- add output to ping tasks (#6)
|
|
16
14
|
|
|
17
15
|
|
|
18
16
|
|
data/README.md
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
[](https://badge.fury.io/rb/indexmap)
|
|
4
4
|
[](https://github.com/ethos-link/indexmap/actions/workflows/ruby.yml)
|
|
5
5
|
|
|
6
|
-
`indexmap` is a small Ruby gem for generating XML
|
|
6
|
+
`indexmap` is a small Ruby gem for generating XML sitemaps from explicit Ruby data.
|
|
7
7
|
|
|
8
8
|
It is designed for Rails apps that want:
|
|
9
9
|
|
|
@@ -12,7 +12,7 @@ It is designed for Rails apps that want:
|
|
|
12
12
|
- first-party rake tasks instead of a large DSL
|
|
13
13
|
- easy extraction of sitemap logic into app-owned manifests
|
|
14
14
|
|
|
15
|
-
|
|
15
|
+
By default, `indexmap` writes a sitemap index plus one or more child sitemap files. For simpler sites, it also supports `:single_file` mode, which writes a single `urlset` directly to `sitemap.xml`.
|
|
16
16
|
|
|
17
17
|
## Installation
|
|
18
18
|
|
|
@@ -34,7 +34,7 @@ Or install it directly:
|
|
|
34
34
|
gem install indexmap
|
|
35
35
|
```
|
|
36
36
|
|
|
37
|
-
## Ruby
|
|
37
|
+
## Ruby Usage
|
|
38
38
|
|
|
39
39
|
```ruby
|
|
40
40
|
require "indexmap"
|
|
@@ -56,7 +56,7 @@ Indexmap::Writer.new(
|
|
|
56
56
|
).write
|
|
57
57
|
```
|
|
58
58
|
|
|
59
|
-
## Rails
|
|
59
|
+
## Rails Usage
|
|
60
60
|
|
|
61
61
|
In an initializer:
|
|
62
62
|
|
|
@@ -77,14 +77,24 @@ Indexmap.configure do |config|
|
|
|
77
77
|
end
|
|
78
78
|
```
|
|
79
79
|
|
|
80
|
-
|
|
80
|
+
Then run:
|
|
81
81
|
|
|
82
82
|
```bash
|
|
83
83
|
bin/rails sitemap:create
|
|
84
84
|
bin/rails sitemap:format
|
|
85
|
+
bin/rails sitemap:validate
|
|
85
86
|
```
|
|
86
87
|
|
|
87
|
-
|
|
88
|
+
`sitemap:create` is the main task. It writes sitemap files, formats them, and validates the result.
|
|
89
|
+
|
|
90
|
+
### Default Index Mode
|
|
91
|
+
|
|
92
|
+
This is the default behavior. `indexmap` writes:
|
|
93
|
+
|
|
94
|
+
- `public/sitemap.xml` as a sitemap index
|
|
95
|
+
- one or more child sitemap files from `config.sections`
|
|
96
|
+
|
|
97
|
+
### Single-File Mode
|
|
88
98
|
|
|
89
99
|
For sites that only want one `public/sitemap.xml` file:
|
|
90
100
|
|
|
@@ -102,9 +112,9 @@ Indexmap.configure do |config|
|
|
|
102
112
|
end
|
|
103
113
|
```
|
|
104
114
|
|
|
105
|
-
In `:single_file` mode, `indexmap` writes a `urlset` directly to `sitemap.xml
|
|
115
|
+
In `:single_file` mode, `indexmap` writes a `urlset` directly to `sitemap.xml` and reads entries from `config.entries` instead of `config.sections`.
|
|
106
116
|
|
|
107
|
-
## Validation
|
|
117
|
+
## Validation And Parsing
|
|
108
118
|
|
|
109
119
|
`indexmap` also includes small utilities for working with generated sitemap files:
|
|
110
120
|
|
|
@@ -124,27 +134,72 @@ The built-in validator checks for:
|
|
|
124
134
|
|
|
125
135
|
## Search Engine Ping
|
|
126
136
|
|
|
127
|
-
|
|
137
|
+
`indexmap` can ping Google Search Console and IndexNow after sitemap generation.
|
|
138
|
+
|
|
139
|
+
Available rake tasks:
|
|
140
|
+
|
|
141
|
+
```bash
|
|
142
|
+
bin/rails sitemap:validate
|
|
143
|
+
bin/rails sitemap:google:ping
|
|
144
|
+
bin/rails sitemap:index_now:ping
|
|
145
|
+
bin/rails sitemap:index_now:write_key
|
|
146
|
+
bin/rails sitemap:ping
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
### Google Search Console
|
|
150
|
+
|
|
151
|
+
Google pinging requires service account credentials:
|
|
128
152
|
|
|
129
153
|
```ruby
|
|
130
154
|
Indexmap.configure do |config|
|
|
131
155
|
config.google.credentials = -> { ENV["GOOGLE_SITEMAP"] }
|
|
156
|
+
end
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
If `config.google.credentials` is blank, `sitemap:google:ping` skips Google submission.
|
|
160
|
+
|
|
161
|
+
You can optionally override the Search Console property identifier:
|
|
162
|
+
|
|
163
|
+
```ruby
|
|
164
|
+
Indexmap.configure do |config|
|
|
165
|
+
config.google.credentials = -> { ENV["GOOGLE_SITEMAP"] }
|
|
166
|
+
config.google.property = -> { "sc-domain:example.com" }
|
|
167
|
+
end
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
If `config.google.property` is not set, `indexmap` defaults to `sc-domain:<host>`.
|
|
171
|
+
|
|
172
|
+
### IndexNow
|
|
173
|
+
|
|
174
|
+
IndexNow submission requires a key. `indexmap` supports two ways to provide it:
|
|
175
|
+
|
|
176
|
+
- set `config.index_now.key`
|
|
177
|
+
- or keep a valid verification file at `public/<key>.txt`
|
|
178
|
+
|
|
179
|
+
Configured-key example:
|
|
180
|
+
|
|
181
|
+
```ruby
|
|
182
|
+
Indexmap.configure do |config|
|
|
132
183
|
config.index_now.key = -> { ENV["INDEXNOW_KEY"] }
|
|
133
184
|
end
|
|
134
185
|
```
|
|
135
186
|
|
|
136
|
-
|
|
187
|
+
If `config.index_now.key` is set, `sitemap:create` also writes the matching `public/<key>.txt` verification file automatically.
|
|
137
188
|
|
|
138
|
-
|
|
189
|
+
If you prefer the file-based flow, run:
|
|
139
190
|
|
|
140
191
|
```bash
|
|
141
|
-
bin/rails sitemap:validate
|
|
142
|
-
bin/rails sitemap:google:ping
|
|
143
|
-
bin/rails sitemap:index_now:ping
|
|
144
|
-
bin/rails sitemap:ping
|
|
145
192
|
bin/rails sitemap:index_now:write_key
|
|
146
193
|
```
|
|
147
194
|
|
|
195
|
+
That task:
|
|
196
|
+
|
|
197
|
+
- reuses an existing valid key file when present
|
|
198
|
+
- otherwise generates a new key in `public/<key>.txt`
|
|
199
|
+
- makes that key available to `sitemap:index_now:ping` without adding `config.index_now.key`
|
|
200
|
+
|
|
201
|
+
If neither a configured key nor a valid key file is present, `sitemap:index_now:ping` skips IndexNow submission.
|
|
202
|
+
|
|
148
203
|
## Development
|
|
149
204
|
|
|
150
205
|
Run tests:
|
|
@@ -165,11 +220,7 @@ Run the full default task:
|
|
|
165
220
|
bundle exec rake
|
|
166
221
|
```
|
|
167
222
|
|
|
168
|
-
Tests generate a coverage report automatically.
|
|
169
|
-
|
|
170
|
-
```bash
|
|
171
|
-
bundle exec rake test
|
|
172
|
-
```
|
|
223
|
+
Tests generate a coverage report automatically.
|
|
173
224
|
|
|
174
225
|
Note: `Gemfile.lock` is intentionally not tracked for this gem, following normal Ruby library conventions.
|
|
175
226
|
|
|
@@ -21,7 +21,7 @@ module Indexmap
|
|
|
21
21
|
resolve(@key)
|
|
22
22
|
end
|
|
23
23
|
|
|
24
|
-
def key_path(public_path:)
|
|
24
|
+
def key_path(public_path:, key: self.key)
|
|
25
25
|
configured_path = resolve(@key_path)
|
|
26
26
|
return Pathname(configured_path) unless configured_path.to_s.strip.empty?
|
|
27
27
|
return if key.to_s.strip.empty?
|
|
@@ -17,10 +17,13 @@ module Indexmap
|
|
|
17
17
|
def ping
|
|
18
18
|
if google_configuration.credentials.to_s.strip.empty?
|
|
19
19
|
logger.debug("Google sitemap credentials not configured.")
|
|
20
|
-
return
|
|
20
|
+
return {status: :skipped, reason: :missing_credentials}
|
|
21
21
|
end
|
|
22
22
|
|
|
23
|
-
|
|
23
|
+
results = sitemap_files.map { |sitemap_file| ping_sitemap(sitemap_file) }
|
|
24
|
+
return {status: :skipped, reason: :no_sitemaps} if results.empty?
|
|
25
|
+
|
|
26
|
+
summarize_results(results)
|
|
24
27
|
end
|
|
25
28
|
|
|
26
29
|
private
|
|
@@ -36,13 +39,26 @@ module Indexmap
|
|
|
36
39
|
|
|
37
40
|
unless authorized?
|
|
38
41
|
logger.error("Google Search Console does not have access to the site: #{root_domain}")
|
|
39
|
-
return
|
|
42
|
+
return {
|
|
43
|
+
status: :failed,
|
|
44
|
+
reason: :unauthorized,
|
|
45
|
+
property: property_identifier,
|
|
46
|
+
root_domain: root_domain
|
|
47
|
+
}
|
|
40
48
|
end
|
|
41
49
|
|
|
42
50
|
webmasters_service.submit_sitemap(property_identifier, sitemap_url)
|
|
43
51
|
logger.debug { "Successfully pinged Google with sitemap: #{sitemap_url}" }
|
|
52
|
+
{status: :submitted, sitemap_url: sitemap_url}
|
|
44
53
|
rescue ::Google::Apis::ClientError => e
|
|
45
54
|
logger.debug { "Failed to ping Google for #{sitemap_url}. Status: #{e.status_code}, Body: #{e.body}" }
|
|
55
|
+
{
|
|
56
|
+
status: :failed,
|
|
57
|
+
reason: :client_error,
|
|
58
|
+
sitemap_url: sitemap_url,
|
|
59
|
+
status_code: e.status_code,
|
|
60
|
+
body: e.body
|
|
61
|
+
}
|
|
46
62
|
end
|
|
47
63
|
|
|
48
64
|
def authorized?
|
|
@@ -73,6 +89,21 @@ module Indexmap
|
|
|
73
89
|
scope: scope
|
|
74
90
|
)
|
|
75
91
|
end
|
|
92
|
+
|
|
93
|
+
def summarize_results(results)
|
|
94
|
+
submitted = results.select { |result| result[:status] == :submitted }
|
|
95
|
+
failures = results.select { |result| result[:status] == :failed }
|
|
96
|
+
|
|
97
|
+
return {status: :submitted, sitemap_count: submitted.count, submitted: submitted} if failures.empty?
|
|
98
|
+
return {status: :failed, sitemap_count: 0, failures: failures} if submitted.empty?
|
|
99
|
+
|
|
100
|
+
{
|
|
101
|
+
status: :partial,
|
|
102
|
+
sitemap_count: submitted.count,
|
|
103
|
+
submitted: submitted,
|
|
104
|
+
failures: failures
|
|
105
|
+
}
|
|
106
|
+
end
|
|
76
107
|
end
|
|
77
108
|
end
|
|
78
109
|
end
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
require "faraday"
|
|
4
4
|
require "json"
|
|
5
|
+
require "securerandom"
|
|
5
6
|
require "time"
|
|
6
7
|
|
|
7
8
|
module Indexmap
|
|
@@ -16,37 +17,49 @@ module Indexmap
|
|
|
16
17
|
api_key = read_api_key
|
|
17
18
|
unless api_key
|
|
18
19
|
logger.debug("IndexNow API key is not configured.")
|
|
19
|
-
return
|
|
20
|
+
return {status: :skipped, reason: :missing_key}
|
|
20
21
|
end
|
|
21
22
|
|
|
22
23
|
entries = entries_to_ping
|
|
23
24
|
if entries.empty?
|
|
24
25
|
logger.debug("IndexNow: no URLs matched the current filter.")
|
|
25
|
-
return
|
|
26
|
+
return {status: :skipped, reason: :no_urls}
|
|
26
27
|
end
|
|
27
28
|
|
|
28
|
-
entries.each_slice(max_urls_per_request) do |batch|
|
|
29
|
+
results = entries.each_slice(max_urls_per_request).map do |batch|
|
|
29
30
|
urls = batch.map(&:loc)
|
|
30
31
|
|
|
31
32
|
if dry_run?
|
|
32
33
|
logger.debug { "IndexNow dry-run: would ping #{urls.count} URLs." }
|
|
33
|
-
next
|
|
34
|
+
next({status: :dry_run, url_count: urls.count})
|
|
34
35
|
end
|
|
35
36
|
|
|
36
37
|
submit_batch(api_key: api_key, urls: urls)
|
|
37
38
|
end
|
|
39
|
+
|
|
40
|
+
summarize_results(results)
|
|
38
41
|
end
|
|
39
42
|
|
|
40
|
-
def write_key_file
|
|
41
|
-
key =
|
|
43
|
+
def write_key_file(key: index_now_configuration.key, path: nil)
|
|
44
|
+
key = key.to_s.strip
|
|
42
45
|
return if key.empty?
|
|
43
46
|
|
|
44
|
-
path
|
|
47
|
+
path ||= index_now_configuration.key_path(public_path: configuration.public_path, key: key)
|
|
45
48
|
FileUtils.mkdir_p(path.dirname)
|
|
46
49
|
File.write(path, "#{key}\n")
|
|
47
50
|
path
|
|
48
51
|
end
|
|
49
52
|
|
|
53
|
+
def ensure_key_file
|
|
54
|
+
configured_key = index_now_configuration.key.to_s.strip
|
|
55
|
+
return write_key_file(key: configured_key) unless configured_key.empty?
|
|
56
|
+
|
|
57
|
+
return existing_key_file if existing_key_file
|
|
58
|
+
|
|
59
|
+
key = generated_key
|
|
60
|
+
write_key_file(key: key, path: configuration.public_path.join("#{key}.txt"))
|
|
61
|
+
end
|
|
62
|
+
|
|
50
63
|
private
|
|
51
64
|
|
|
52
65
|
attr_reader :connection
|
|
@@ -137,10 +150,10 @@ module Indexmap
|
|
|
137
150
|
|
|
138
151
|
if response.success?
|
|
139
152
|
logger.debug { "Successfully pinged IndexNow with #{urls.count} URLs." }
|
|
140
|
-
|
|
153
|
+
{status: :submitted, url_count: urls.count}
|
|
141
154
|
else
|
|
142
155
|
logger.debug { "Failed to ping IndexNow. Status: #{response.status}, Body: #{response.body}" }
|
|
143
|
-
|
|
156
|
+
{status: :failed, url_count: urls.count, status_code: response.status, body: response.body}
|
|
144
157
|
end
|
|
145
158
|
end
|
|
146
159
|
|
|
@@ -158,15 +171,65 @@ module Indexmap
|
|
|
158
171
|
configured_key = index_now_configuration.key.to_s.strip
|
|
159
172
|
return configured_key unless configured_key.empty?
|
|
160
173
|
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
174
|
+
existing_key_file&.read&.strip
|
|
175
|
+
end
|
|
176
|
+
|
|
177
|
+
def existing_key_file
|
|
178
|
+
configured_path = index_now_configuration.key_path(public_path: configuration.public_path)
|
|
179
|
+
return configured_path if valid_key_file?(configured_path)
|
|
180
|
+
|
|
181
|
+
configuration.public_path.glob("*.txt").find { |file| valid_key_file?(file) }
|
|
182
|
+
end
|
|
183
|
+
|
|
184
|
+
def valid_key_file?(path)
|
|
185
|
+
return false unless path&.file?
|
|
186
|
+
|
|
187
|
+
filename = path.basename(".txt").to_s
|
|
188
|
+
return false unless filename.match?(/\A[a-zA-Z0-9-]{8,128}\z/)
|
|
189
|
+
|
|
190
|
+
path.read.strip == filename
|
|
191
|
+
end
|
|
192
|
+
|
|
193
|
+
def generated_key
|
|
194
|
+
SecureRandom.uuid
|
|
195
|
+
end
|
|
196
|
+
|
|
197
|
+
def summarize_results(results)
|
|
198
|
+
dry_runs = results.select { |result| result[:status] == :dry_run }
|
|
199
|
+
submitted = results.select { |result| result[:status] == :submitted }
|
|
200
|
+
failures = results.select { |result| result[:status] == :failed }
|
|
201
|
+
|
|
202
|
+
if dry_runs.any?
|
|
203
|
+
return {
|
|
204
|
+
status: :dry_run,
|
|
205
|
+
url_count: dry_runs.sum { |result| result[:url_count] },
|
|
206
|
+
batch_count: dry_runs.count
|
|
207
|
+
}
|
|
208
|
+
end
|
|
209
|
+
|
|
210
|
+
if failures.empty?
|
|
211
|
+
return {
|
|
212
|
+
status: :submitted,
|
|
213
|
+
url_count: submitted.sum { |result| result[:url_count] },
|
|
214
|
+
batch_count: submitted.count
|
|
215
|
+
}
|
|
216
|
+
end
|
|
164
217
|
|
|
165
|
-
|
|
218
|
+
if submitted.empty?
|
|
219
|
+
return {
|
|
220
|
+
status: :failed,
|
|
221
|
+
url_count: 0,
|
|
222
|
+
batch_count: 0,
|
|
223
|
+
failures: failures
|
|
224
|
+
}
|
|
166
225
|
end
|
|
167
|
-
return nil unless key_file
|
|
168
226
|
|
|
169
|
-
|
|
227
|
+
{
|
|
228
|
+
status: :partial,
|
|
229
|
+
url_count: submitted.sum { |result| result[:url_count] },
|
|
230
|
+
batch_count: submitted.count,
|
|
231
|
+
failures: failures
|
|
232
|
+
}
|
|
170
233
|
end
|
|
171
234
|
end
|
|
172
235
|
end
|
data/lib/indexmap/task_runner.rb
CHANGED
|
@@ -11,7 +11,7 @@ module Indexmap
|
|
|
11
11
|
def create
|
|
12
12
|
remove_existing_sitemap_files
|
|
13
13
|
configuration.writer.write
|
|
14
|
-
write_index_now_key
|
|
14
|
+
{files: sitemap_files, index_now_key_path: write_index_now_key}
|
|
15
15
|
end
|
|
16
16
|
|
|
17
17
|
def format
|
|
@@ -27,28 +27,38 @@ module Indexmap
|
|
|
27
27
|
|
|
28
28
|
File.write(file_path, document.to_xml(indent: 2, save_with: save_options))
|
|
29
29
|
end
|
|
30
|
+
|
|
31
|
+
sitemap_files
|
|
30
32
|
end
|
|
31
33
|
|
|
32
34
|
def validate
|
|
33
35
|
Validator.new(configuration: configuration).validate!
|
|
36
|
+
sitemap_files
|
|
34
37
|
end
|
|
35
38
|
|
|
36
|
-
def write_index_now_key
|
|
37
|
-
Pinger::IndexNow.new(configuration: configuration)
|
|
39
|
+
def write_index_now_key(generate_if_missing: false)
|
|
40
|
+
pinger = Pinger::IndexNow.new(configuration: configuration)
|
|
41
|
+
return pinger.ensure_key_file if generate_if_missing
|
|
42
|
+
|
|
43
|
+
pinger.write_key_file
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
def public_path
|
|
47
|
+
configuration.public_path
|
|
38
48
|
end
|
|
39
49
|
|
|
40
50
|
private
|
|
41
51
|
|
|
42
52
|
attr_reader :configuration
|
|
43
53
|
|
|
44
|
-
def sitemap_files
|
|
45
|
-
Dir.glob(configuration.public_path.join("sitemap*.xml"))
|
|
46
|
-
end
|
|
47
|
-
|
|
48
54
|
def remove_existing_sitemap_files
|
|
49
55
|
Dir.glob(configuration.public_path.join("sitemap*.xml*")).each do |file_path|
|
|
50
56
|
File.delete(file_path)
|
|
51
57
|
end
|
|
52
58
|
end
|
|
59
|
+
|
|
60
|
+
def sitemap_files
|
|
61
|
+
Dir.glob(configuration.public_path.join("sitemap*.xml")).sort
|
|
62
|
+
end
|
|
53
63
|
end
|
|
54
64
|
end
|
data/lib/indexmap/version.rb
CHANGED
|
@@ -2,19 +2,28 @@ namespace :sitemap do
|
|
|
2
2
|
desc "Create sitemap files"
|
|
3
3
|
task create: :environment do
|
|
4
4
|
runner = Indexmap::TaskRunner.new
|
|
5
|
-
runner.create
|
|
5
|
+
create_result = runner.create
|
|
6
6
|
runner.format
|
|
7
|
-
runner.validate
|
|
7
|
+
validated_files = runner.validate
|
|
8
|
+
|
|
9
|
+
puts "Created, formatted, and validated #{file_count(validated_files)} in #{public_directory(runner)}."
|
|
10
|
+
puts "IndexNow key file: #{create_result[:index_now_key_path]}" if create_result[:index_now_key_path]
|
|
8
11
|
end
|
|
9
12
|
|
|
10
13
|
desc "Format sitemap files for better readability"
|
|
11
14
|
task format: :environment do
|
|
12
|
-
Indexmap::TaskRunner.new
|
|
15
|
+
runner = Indexmap::TaskRunner.new
|
|
16
|
+
formatted_files = runner.format
|
|
17
|
+
|
|
18
|
+
puts "Formatted #{file_count(formatted_files)} in #{public_directory(runner)}."
|
|
13
19
|
end
|
|
14
20
|
|
|
15
21
|
desc "Validate sitemap shape and URL hygiene"
|
|
16
22
|
task validate: :environment do
|
|
17
|
-
Indexmap::TaskRunner.new
|
|
23
|
+
runner = Indexmap::TaskRunner.new
|
|
24
|
+
validated_files = runner.validate
|
|
25
|
+
|
|
26
|
+
puts "Validated #{file_count(validated_files)} for sitemap shape and URL hygiene."
|
|
18
27
|
end
|
|
19
28
|
|
|
20
29
|
desc "Ping all configured search engines"
|
|
@@ -26,20 +35,101 @@ namespace :sitemap do
|
|
|
26
35
|
namespace :google do
|
|
27
36
|
desc "Ping Google Search Console"
|
|
28
37
|
task ping: :environment do
|
|
29
|
-
Indexmap::Pinger::Google.new.ping
|
|
38
|
+
result = Indexmap::Pinger::Google.new.ping
|
|
39
|
+
|
|
40
|
+
case result[:status]
|
|
41
|
+
when :submitted
|
|
42
|
+
puts "Submitted #{result[:sitemap_count]} sitemap #{(result[:sitemap_count] == 1) ? "file" : "files"} to Google Search Console."
|
|
43
|
+
when :partial
|
|
44
|
+
puts "Submitted #{result[:sitemap_count]} sitemap #{(result[:sitemap_count] == 1) ? "file" : "files"} to Google Search Console, with #{result[:failures].count} failure#{"s" unless result[:failures].count == 1}."
|
|
45
|
+
result[:failures].each { |failure| puts format_google_ping_failure(failure) }
|
|
46
|
+
when :failed
|
|
47
|
+
result[:failures].each { |failure| puts format_google_ping_failure(failure) }
|
|
48
|
+
when :skipped
|
|
49
|
+
puts format_google_ping_skip(result)
|
|
50
|
+
end
|
|
30
51
|
end
|
|
31
52
|
end
|
|
32
53
|
|
|
33
54
|
namespace :index_now do
|
|
34
55
|
desc "Ping IndexNow. ENV: SINCE=2026-04-18T10:30:00Z or INDEXNOW_RECENT_HOURS=24"
|
|
35
56
|
task ping: :environment do
|
|
36
|
-
Indexmap::Pinger::IndexNow.new.ping
|
|
57
|
+
result = Indexmap::Pinger::IndexNow.new.ping
|
|
58
|
+
|
|
59
|
+
case result[:status]
|
|
60
|
+
when :submitted
|
|
61
|
+
puts "Submitted #{result[:url_count]} URL#{"s" unless result[:url_count] == 1} to IndexNow in #{result[:batch_count]} request#{"s" unless result[:batch_count] == 1}."
|
|
62
|
+
when :partial
|
|
63
|
+
puts "Submitted #{result[:url_count]} URL#{"s" unless result[:url_count] == 1} to IndexNow in #{result[:batch_count]} request#{"s" unless result[:batch_count] == 1}, with #{result[:failures].count} failure#{"s" unless result[:failures].count == 1}."
|
|
64
|
+
result[:failures].each { |failure| puts format_index_now_ping_failure(failure) }
|
|
65
|
+
when :failed
|
|
66
|
+
result[:failures].each { |failure| puts format_index_now_ping_failure(failure) }
|
|
67
|
+
when :dry_run
|
|
68
|
+
puts "IndexNow dry-run: would submit #{result[:url_count]} URL#{"s" unless result[:url_count] == 1} in #{result[:batch_count]} request#{"s" unless result[:batch_count] == 1}."
|
|
69
|
+
when :skipped
|
|
70
|
+
puts format_index_now_ping_skip(result)
|
|
71
|
+
end
|
|
37
72
|
end
|
|
38
73
|
|
|
39
|
-
desc "
|
|
74
|
+
desc "Ensure the IndexNow key file exists in public/"
|
|
40
75
|
task write_key: :environment do
|
|
41
|
-
path = Indexmap::TaskRunner.new.write_index_now_key
|
|
42
|
-
|
|
76
|
+
path = Indexmap::TaskRunner.new.write_index_now_key(generate_if_missing: true)
|
|
77
|
+
if path
|
|
78
|
+
puts "IndexNow key file available at #{path}."
|
|
79
|
+
else
|
|
80
|
+
puts "IndexNow key is not configured; skipped key file write."
|
|
81
|
+
end
|
|
82
|
+
end
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
def file_count(files)
|
|
86
|
+
count = Array(files).size
|
|
87
|
+
"#{count} sitemap #{(count == 1) ? "file" : "files"}"
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
def public_directory(runner)
|
|
91
|
+
runner.public_path
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
def format_google_ping_failure(failure)
|
|
95
|
+
case failure[:reason]
|
|
96
|
+
when :unauthorized
|
|
97
|
+
"Google Search Console does not have access to #{failure[:property]}."
|
|
98
|
+
when :client_error
|
|
99
|
+
"Google ping failed for #{failure[:sitemap_url]} (status #{failure[:status_code]})."
|
|
100
|
+
else
|
|
101
|
+
"Google ping failed."
|
|
102
|
+
end
|
|
103
|
+
end
|
|
104
|
+
|
|
105
|
+
def format_google_ping_skip(result)
|
|
106
|
+
case result[:reason]
|
|
107
|
+
when :missing_credentials
|
|
108
|
+
"Google sitemap credentials are not configured; skipped Google submission."
|
|
109
|
+
when :no_sitemaps
|
|
110
|
+
"No sitemap files found; skipped Google submission."
|
|
111
|
+
else
|
|
112
|
+
"Skipped Google submission."
|
|
113
|
+
end
|
|
114
|
+
end
|
|
115
|
+
|
|
116
|
+
def format_index_now_ping_failure(failure)
|
|
117
|
+
case failure[:status_code]
|
|
118
|
+
when nil
|
|
119
|
+
"IndexNow submission failed."
|
|
120
|
+
else
|
|
121
|
+
"IndexNow submission failed for #{failure[:url_count]} URL#{"s" unless failure[:url_count] == 1} (status #{failure[:status_code]})."
|
|
122
|
+
end
|
|
123
|
+
end
|
|
124
|
+
|
|
125
|
+
def format_index_now_ping_skip(result)
|
|
126
|
+
case result[:reason]
|
|
127
|
+
when :missing_key
|
|
128
|
+
"IndexNow key is not configured and no valid key file was found; skipped IndexNow submission."
|
|
129
|
+
when :no_urls
|
|
130
|
+
"No sitemap URLs matched the current IndexNow filter; skipped IndexNow submission."
|
|
131
|
+
else
|
|
132
|
+
"Skipped IndexNow submission."
|
|
43
133
|
end
|
|
44
134
|
end
|
|
45
135
|
end
|
|
@@ -40,7 +40,7 @@ class IndexmapPingerGoogleTest < Minitest::Test
|
|
|
40
40
|
:fake_authorizer
|
|
41
41
|
end
|
|
42
42
|
|
|
43
|
-
Indexmap::Pinger::Google.new(
|
|
43
|
+
result = Indexmap::Pinger::Google.new(
|
|
44
44
|
configuration: configuration,
|
|
45
45
|
service: service,
|
|
46
46
|
credentials_builder: credentials_builder
|
|
@@ -49,6 +49,8 @@ class IndexmapPingerGoogleTest < Minitest::Test
|
|
|
49
49
|
assert_equal [["{\"type\":\"service_account\"}", "https://www.googleapis.com/auth/webmasters"]], builder_calls
|
|
50
50
|
assert_equal :fake_authorizer, service.authorization
|
|
51
51
|
assert_equal ["sc-domain:example.com", "https://www.example.com/sitemap.xml"], service.submitted
|
|
52
|
+
assert_equal :submitted, result[:status]
|
|
53
|
+
assert_equal 1, result[:sitemap_count]
|
|
52
54
|
end
|
|
53
55
|
end
|
|
54
56
|
|
|
@@ -63,9 +65,53 @@ class IndexmapPingerGoogleTest < Minitest::Test
|
|
|
63
65
|
|
|
64
66
|
service = FakeWebmastersService.new(site_urls: ["sc-domain:example.com"])
|
|
65
67
|
|
|
66
|
-
Indexmap::Pinger::Google.new(configuration: configuration, service: service).ping
|
|
68
|
+
result = Indexmap::Pinger::Google.new(configuration: configuration, service: service).ping
|
|
67
69
|
|
|
68
70
|
assert_nil service.submitted
|
|
71
|
+
assert_equal({status: :skipped, reason: :missing_credentials}, result)
|
|
72
|
+
end
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
def test_reports_missing_sitemap_files
|
|
76
|
+
Dir.mktmpdir do |dir|
|
|
77
|
+
configuration = Indexmap::Configuration.new
|
|
78
|
+
configuration.base_url = "https://www.example.com"
|
|
79
|
+
configuration.public_path = Pathname(dir)
|
|
80
|
+
configuration.google.credentials = "{\"type\":\"service_account\"}"
|
|
81
|
+
|
|
82
|
+
service = FakeWebmastersService.new(site_urls: ["sc-domain:example.com"])
|
|
83
|
+
result = Indexmap::Pinger::Google.new(
|
|
84
|
+
configuration: configuration,
|
|
85
|
+
service: service,
|
|
86
|
+
credentials_builder: ->(**) { :fake_authorizer }
|
|
87
|
+
).ping
|
|
88
|
+
|
|
89
|
+
assert_equal({status: :skipped, reason: :no_sitemaps}, result)
|
|
90
|
+
assert_nil service.submitted
|
|
91
|
+
end
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
def test_reports_google_authorization_failure
|
|
95
|
+
Dir.mktmpdir do |dir|
|
|
96
|
+
public_path = Pathname(dir)
|
|
97
|
+
public_path.join("sitemap.xml").write("<sitemapindex/>")
|
|
98
|
+
|
|
99
|
+
configuration = Indexmap::Configuration.new
|
|
100
|
+
configuration.base_url = "https://www.example.com"
|
|
101
|
+
configuration.public_path = public_path
|
|
102
|
+
configuration.google.credentials = "{\"type\":\"service_account\"}"
|
|
103
|
+
|
|
104
|
+
service = FakeWebmastersService.new(site_urls: ["sc-domain:not-example.org"])
|
|
105
|
+
result = Indexmap::Pinger::Google.new(
|
|
106
|
+
configuration: configuration,
|
|
107
|
+
service: service,
|
|
108
|
+
credentials_builder: ->(**) { :fake_authorizer }
|
|
109
|
+
).ping
|
|
110
|
+
|
|
111
|
+
assert_equal :failed, result[:status]
|
|
112
|
+
assert_equal 1, result[:failures].count
|
|
113
|
+
assert_equal :unauthorized, result[:failures].first[:reason]
|
|
114
|
+
assert_nil service.submitted
|
|
69
115
|
end
|
|
70
116
|
end
|
|
71
117
|
end
|
|
@@ -17,6 +17,49 @@ class IndexmapPingerIndexNowTest < Minitest::Test
|
|
|
17
17
|
end
|
|
18
18
|
end
|
|
19
19
|
|
|
20
|
+
def test_ensure_key_file_generates_a_key_when_configuration_is_missing
|
|
21
|
+
Dir.mktmpdir do |dir|
|
|
22
|
+
configuration = Indexmap::Configuration.new
|
|
23
|
+
configuration.base_url = "https://www.example.com"
|
|
24
|
+
configuration.public_path = Pathname(dir)
|
|
25
|
+
|
|
26
|
+
path = Indexmap::Pinger::IndexNow.new(configuration: configuration).ensure_key_file
|
|
27
|
+
|
|
28
|
+
assert_match(/\A[a-z0-9-]{8,128}\.txt\z/, path.basename.to_s)
|
|
29
|
+
assert_equal "#{path.basename(".txt")}\n", path.read
|
|
30
|
+
end
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
def test_pings_using_existing_key_file_when_key_is_not_configured
|
|
34
|
+
Dir.mktmpdir do |dir|
|
|
35
|
+
public_path = Pathname(dir)
|
|
36
|
+
key_path = public_path.join("test-key-123.txt")
|
|
37
|
+
key_path.write("test-key-123\n")
|
|
38
|
+
write_sitemap_files(
|
|
39
|
+
public_path,
|
|
40
|
+
marketing_lastmod: "2026-04-18T00:00:00Z",
|
|
41
|
+
insights_lastmod: "2026-04-10T00:00:00Z"
|
|
42
|
+
)
|
|
43
|
+
|
|
44
|
+
configuration = Indexmap::Configuration.new
|
|
45
|
+
configuration.base_url = "https://www.example.com"
|
|
46
|
+
configuration.public_path = public_path
|
|
47
|
+
|
|
48
|
+
indexnow_url = "https://api.indexnow.org/indexnow"
|
|
49
|
+
stub_request(:post, indexnow_url).to_return(status: 200, body: "", headers: {})
|
|
50
|
+
|
|
51
|
+
result = Indexmap::Pinger::IndexNow.new(configuration: configuration).ping
|
|
52
|
+
|
|
53
|
+
assert_requested(:post, indexnow_url, times: 1) do |request|
|
|
54
|
+
payload = JSON.parse(request.body)
|
|
55
|
+
assert_equal "test-key-123", payload.fetch("key")
|
|
56
|
+
end
|
|
57
|
+
assert_equal :submitted, result[:status]
|
|
58
|
+
assert_equal 2, result[:url_count]
|
|
59
|
+
assert_equal 1, result[:batch_count]
|
|
60
|
+
end
|
|
61
|
+
end
|
|
62
|
+
|
|
20
63
|
def test_pings_all_sitemap_urls_when_no_cutoff_is_provided
|
|
21
64
|
Dir.mktmpdir do |dir|
|
|
22
65
|
public_path = Pathname(dir)
|
|
@@ -34,7 +77,7 @@ class IndexmapPingerIndexNowTest < Minitest::Test
|
|
|
34
77
|
indexnow_url = "https://api.indexnow.org/indexnow"
|
|
35
78
|
stub_request(:post, indexnow_url).to_return(status: 200, body: "", headers: {})
|
|
36
79
|
|
|
37
|
-
Indexmap::Pinger::IndexNow.new(configuration: configuration).ping
|
|
80
|
+
result = Indexmap::Pinger::IndexNow.new(configuration: configuration).ping
|
|
38
81
|
|
|
39
82
|
assert_requested(:post, indexnow_url, times: 1) do |request|
|
|
40
83
|
payload = JSON.parse(request.body)
|
|
@@ -43,6 +86,9 @@ class IndexmapPingerIndexNowTest < Minitest::Test
|
|
|
43
86
|
"https://www.example.com/insights/us/restaurants/overview"
|
|
44
87
|
].sort, payload.fetch("urlList").sort
|
|
45
88
|
end
|
|
89
|
+
assert_equal :submitted, result[:status]
|
|
90
|
+
assert_equal 2, result[:url_count]
|
|
91
|
+
assert_equal 1, result[:batch_count]
|
|
46
92
|
end
|
|
47
93
|
end
|
|
48
94
|
|
|
@@ -64,7 +110,11 @@ class IndexmapPingerIndexNowTest < Minitest::Test
|
|
|
64
110
|
stub_request(:post, indexnow_url).to_return(status: 200, body: "", headers: {})
|
|
65
111
|
|
|
66
112
|
with_env("SINCE" => "2026-04-15T00:00:00Z") do
|
|
67
|
-
Indexmap::Pinger::IndexNow.new(configuration: configuration).ping
|
|
113
|
+
result = Indexmap::Pinger::IndexNow.new(configuration: configuration).ping
|
|
114
|
+
|
|
115
|
+
assert_equal :submitted, result[:status]
|
|
116
|
+
assert_equal 1, result[:url_count]
|
|
117
|
+
assert_equal 1, result[:batch_count]
|
|
68
118
|
end
|
|
69
119
|
|
|
70
120
|
assert_requested(:post, indexnow_url, times: 1) do |request|
|
|
@@ -74,6 +124,74 @@ class IndexmapPingerIndexNowTest < Minitest::Test
|
|
|
74
124
|
end
|
|
75
125
|
end
|
|
76
126
|
|
|
127
|
+
def test_skips_indexnow_ping_when_key_is_missing
|
|
128
|
+
Dir.mktmpdir do |dir|
|
|
129
|
+
public_path = Pathname(dir)
|
|
130
|
+
write_sitemap_files(
|
|
131
|
+
public_path,
|
|
132
|
+
marketing_lastmod: "2026-04-18T00:00:00Z",
|
|
133
|
+
insights_lastmod: "2026-04-10T00:00:00Z"
|
|
134
|
+
)
|
|
135
|
+
|
|
136
|
+
configuration = Indexmap::Configuration.new
|
|
137
|
+
configuration.base_url = "https://www.example.com"
|
|
138
|
+
configuration.public_path = public_path
|
|
139
|
+
|
|
140
|
+
result = Indexmap::Pinger::IndexNow.new(configuration: configuration).ping
|
|
141
|
+
|
|
142
|
+
assert_equal({status: :skipped, reason: :missing_key}, result)
|
|
143
|
+
end
|
|
144
|
+
end
|
|
145
|
+
|
|
146
|
+
def test_reports_indexnow_dry_run
|
|
147
|
+
Dir.mktmpdir do |dir|
|
|
148
|
+
public_path = Pathname(dir)
|
|
149
|
+
write_sitemap_files(
|
|
150
|
+
public_path,
|
|
151
|
+
marketing_lastmod: "2026-04-18T00:00:00Z",
|
|
152
|
+
insights_lastmod: "2026-04-10T00:00:00Z"
|
|
153
|
+
)
|
|
154
|
+
|
|
155
|
+
configuration = Indexmap::Configuration.new
|
|
156
|
+
configuration.base_url = "https://www.example.com"
|
|
157
|
+
configuration.public_path = public_path
|
|
158
|
+
configuration.index_now.key = "test-key"
|
|
159
|
+
|
|
160
|
+
with_env("INDEXNOW_DRY_RUN" => "1") do
|
|
161
|
+
result = Indexmap::Pinger::IndexNow.new(configuration: configuration).ping
|
|
162
|
+
|
|
163
|
+
assert_equal :dry_run, result[:status]
|
|
164
|
+
assert_equal 2, result[:url_count]
|
|
165
|
+
assert_equal 1, result[:batch_count]
|
|
166
|
+
end
|
|
167
|
+
end
|
|
168
|
+
end
|
|
169
|
+
|
|
170
|
+
def test_reports_failed_indexnow_submission
|
|
171
|
+
Dir.mktmpdir do |dir|
|
|
172
|
+
public_path = Pathname(dir)
|
|
173
|
+
write_sitemap_files(
|
|
174
|
+
public_path,
|
|
175
|
+
marketing_lastmod: "2026-04-18T00:00:00Z",
|
|
176
|
+
insights_lastmod: "2026-04-10T00:00:00Z"
|
|
177
|
+
)
|
|
178
|
+
|
|
179
|
+
configuration = Indexmap::Configuration.new
|
|
180
|
+
configuration.base_url = "https://www.example.com"
|
|
181
|
+
configuration.public_path = public_path
|
|
182
|
+
configuration.index_now.key = "test-key"
|
|
183
|
+
|
|
184
|
+
indexnow_url = "https://api.indexnow.org/indexnow"
|
|
185
|
+
stub_request(:post, indexnow_url).to_return(status: 500, body: "boom", headers: {})
|
|
186
|
+
|
|
187
|
+
result = Indexmap::Pinger::IndexNow.new(configuration: configuration).ping
|
|
188
|
+
|
|
189
|
+
assert_equal :failed, result[:status]
|
|
190
|
+
assert_equal 1, result[:failures].count
|
|
191
|
+
assert_equal 500, result[:failures].first[:status_code]
|
|
192
|
+
end
|
|
193
|
+
end
|
|
194
|
+
|
|
77
195
|
private
|
|
78
196
|
|
|
79
197
|
def with_env(overrides)
|
|
@@ -20,11 +20,13 @@ class IndexmapTaskRunnerTest < Minitest::Test
|
|
|
20
20
|
]
|
|
21
21
|
configuration.index_now.key = "test-key"
|
|
22
22
|
|
|
23
|
-
Indexmap::TaskRunner.new(configuration: configuration).create
|
|
23
|
+
result = Indexmap::TaskRunner.new(configuration: configuration).create
|
|
24
24
|
|
|
25
25
|
assert_equal false, public_path.join("sitemap-pages.xml.gz").exist?
|
|
26
26
|
assert_includes public_path.join("sitemap.xml").read, "<sitemapindex"
|
|
27
27
|
assert_equal "test-key\n", public_path.join("test-key.txt").read
|
|
28
|
+
assert_equal [public_path.join("sitemap-pages.xml").to_s, public_path.join("sitemap.xml").to_s], result[:files]
|
|
29
|
+
assert_equal public_path.join("test-key.txt"), result[:index_now_key_path]
|
|
28
30
|
end
|
|
29
31
|
end
|
|
30
32
|
|
|
@@ -45,4 +47,17 @@ class IndexmapTaskRunnerTest < Minitest::Test
|
|
|
45
47
|
assert_nil result
|
|
46
48
|
end
|
|
47
49
|
end
|
|
50
|
+
|
|
51
|
+
def test_write_index_now_key_can_generate_a_key_when_requested
|
|
52
|
+
Dir.mktmpdir do |dir|
|
|
53
|
+
configuration = Indexmap::Configuration.new
|
|
54
|
+
configuration.base_url = "https://example.com"
|
|
55
|
+
configuration.public_path = Pathname(dir)
|
|
56
|
+
|
|
57
|
+
result = Indexmap::TaskRunner.new(configuration: configuration).write_index_now_key(generate_if_missing: true)
|
|
58
|
+
|
|
59
|
+
assert_match(/\A[a-z0-9-]{8,128}\.txt\z/, result.basename.to_s)
|
|
60
|
+
assert_equal "#{result.basename(".txt")}\n", result.read
|
|
61
|
+
end
|
|
62
|
+
end
|
|
48
63
|
end
|