index_now 0.1.0 → 0.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 131220889a57d1c72c46f1332899034547b103c7b9594d06de1b1f06dd3937d4
4
- data.tar.gz: a78d03b12c476c130d364ad1d2a91ff9cfcbe3b1928994570730157d7db6dffc
3
+ metadata.gz: 06ba8a7a0e186ca3894a88700f922500577983a59032d14095096422a75edcf5
4
+ data.tar.gz: 4bf6d4c0a6901c2a32d3183158b958a86fdf38bb812718e357390ff69c876024
5
5
  SHA512:
6
- metadata.gz: 36d2dce172e838142c225200bb726dfdd72da7b0a14e11f53576583474ebf0dbe2d627c009f11dd39178ed424792cfa91c723908b193d61fd7faf2c240714b5c
7
- data.tar.gz: 96f998799a9fdf85f2af8f5aa5228f084e8d299708892756b2b4f24e7eda6e4ca55749fca322e0e70c133f79f71c6c3b34d5baede34ee30b54367f2ecec9b196
6
+ metadata.gz: 9e82402c55234ccc653f40cfa029dfba1ef5e4f898350a631d89ba4023fe54686b26c004e82d886795e55fd99cc8a73e49c7ae56bf06c22f8826c2d21e7861d3
7
+ data.tar.gz: c874332fca8db046f2dea57925e1ef4e129eaab7a3c517f55f2271536b420794c5d89e51ebaff1bf410221245faf0c5d36bd134c496328df2b1f93ed3bdd5c88
data/CHANGELOG.md CHANGED
@@ -1,5 +1,20 @@
1
1
  # Changelog
2
2
 
3
+ ## [0.2.1] — 2026-11-02
4
+
5
+ - `index_now announce` no longer needs ActiveSupport to find the key file;
6
+ 0.2.0 failed with ``undefined method `sole'`` outside Rails.
7
+
8
+ ## [0.2.0] — 2026-11-02
9
+
10
+ - Static sites: an `index_now` command (`init`, `pending`, `announce`,
11
+ `announce --all`) reads the built sitemap and records what it sent in
12
+ `.index_now.json`, with the same rule as the Rails engine.
13
+ - Rails is no longer a dependency; the engine loads only when Rails is present.
14
+ - `IndexNow::Client.submit!` takes the key as `key:`.
15
+ - The migration ships only as the install generator's template, so
16
+ `index_now:install:migrations` can no longer add a second copy.
17
+
3
18
  ## [0.1.0] — 2026-10-28
4
19
 
5
20
  - First release: announces sitemap URLs whose `lastmod` moved to IndexNow,
data/README.md CHANGED
@@ -1,9 +1,9 @@
1
1
  # index_now
2
2
 
3
- Tells search engines which pages of a Rails app changed, through
3
+ Tells search engines which pages of a Rails app or a static site changed, through
4
4
  [IndexNow](https://www.indexnow.org), so they recrawl them without waiting.
5
5
 
6
- It reads the app's own sitemap, announces every URL whose `<lastmod>` moved
6
+ It reads the site's own sitemap, announces every URL whose `<lastmod>` moved
7
7
  since it last announced that URL, and records what it sent. Running it when
8
8
  nothing changed sends nothing, so it is safe to call as often as you like.
9
9
 
@@ -11,7 +11,11 @@ nothing changed sends nothing, so it is safe to call as often as you like.
11
11
  also backs ChatGPT search and Copilot), Yandex, Seznam, Naver and Yep.
12
12
  **Google is not an IndexNow participant**; nothing here affects Google.
13
13
 
14
- ## Install
14
+ A Rails app gets an engine that records announcements in its database; a
15
+ static site gets an `index_now` command that records them in a JSON file
16
+ ([Static sites](#static-sites)). Both follow [the rule](#the-rule).
17
+
18
+ ## Install in a Rails app
15
19
 
16
20
  ```ruby
17
21
  gem "index_now"
@@ -90,12 +94,49 @@ bin/rails index_now:announce # send it; exits 1 on failure, says so when di
90
94
  bin/rails index_now:announce_all # send every sitemap URL again
91
95
  ```
92
96
 
97
+ ## Static sites
98
+
99
+ The gem needs no Rails for this; Rails loads only when the app already has it.
100
+
101
+ ```sh
102
+ gem install index_now
103
+ index_now init # writes public/<key>.txt; commit it and deploy it with the site
104
+ index_now pending # what the next announce would send
105
+ index_now announce # send it; exits 1 on failure
106
+ ```
107
+
108
+ `announce` reads the built `public/sitemap.xml` and keeps what it sent in
109
+ `.index_now.json`, one `url: lastmod` pair per URL. Commit that file next to
110
+ the site, so the next run, from any machine, sends only what changed since.
111
+
112
+ Run it at the end of the deploy, once the new pages are live: an engine that
113
+ recrawls a page before the upload, or before a CDN invalidation, reads the old
114
+ one.
115
+
116
+ ```sh
117
+ aws s3 sync public/ "s3://$BUCKET" --delete
118
+ aws cloudfront create-invalidation --distribution-id "$DISTRIBUTION" --paths "/*"
119
+ index_now announce || echo "IndexNow failed; the next deploy will retry"
120
+ ```
121
+
122
+ | Option | Default |
123
+ | --- | --- |
124
+ | `--public DIR` | `public` |
125
+ | `--sitemap PATH` | `<public>/sitemap.xml` |
126
+ | `--state PATH` | `.index_now.json` |
127
+ | `--key KEY` | the `<key>.txt` in `<public>` whose body is its name |
128
+ | `--all` | `announce` sends every URL, not just the changed ones |
129
+
93
130
  ## Rotating the key
94
131
 
95
132
  Rename `public/<old key>.txt` to the new key, put the same key in its body and in
96
133
  `config.key`, deploy, check `https://<host>/<key>.txt` answers 200, then run
97
- `index_now:announce_all`.
134
+ `index_now:announce_all` (`index_now announce --all` on a static site).
98
135
 
99
136
  ## License
100
137
 
101
138
  MIT
139
+
140
+ ## Author
141
+
142
+ Built by [Anton Kopylov](https://www.kopylov.net).
@@ -4,13 +4,7 @@ module IndexNow
4
4
 
5
5
  def self.due(entries)
6
6
  announced = where(url: entries.map(&:url)).pluck(:url, :lastmod).to_h
7
-
8
- entries.select do |entry|
9
- next true unless announced.key?(entry.url)
10
-
11
- previous = announced[entry.url]
12
- entry.lastmod && (previous.nil? || entry.lastmod > previous)
13
- end
7
+ entries.select { |entry| entry.due?(announced) }
14
8
  end
15
9
 
16
10
  def self.record!(entries)
data/exe/index_now ADDED
@@ -0,0 +1,4 @@
1
+ #!/usr/bin/env ruby
2
+ require "index_now/cli"
3
+
4
+ exit IndexNow::CLI.start(ARGV)
@@ -1,6 +1,5 @@
1
1
  require "rails/generators"
2
2
  require "rails/generators/active_record"
3
- require "securerandom"
4
3
 
5
4
  module IndexNow
6
5
  module Generators
@@ -26,8 +25,8 @@ module IndexNow
26
25
 
27
26
  private
28
27
  def key
29
- @key ||= (options[:key] || SecureRandom.hex(16)).tap do |key|
30
- raise ArgumentError, "an IndexNow key is 8 to 128 letters, digits or dashes" unless key.match?(/\A[a-zA-Z0-9-]{8,128}\z/)
28
+ @key ||= (options[:key] || IndexNow::Key.generate).tap do |key|
29
+ raise ArgumentError, "an IndexNow key is 8 to 128 letters, digits or dashes" unless IndexNow::Key.valid?(key)
31
30
  end
32
31
  end
33
32
  end
@@ -0,0 +1,123 @@
1
+ require "optparse"
2
+ require "index_now"
3
+
4
+ module IndexNow
5
+ # `index_now` for a static site: its built sitemap is the input, a JSON file remembers what was announced.
6
+ class CLI
7
+ def self.start(argv, out: $stdout, err: $stderr) = new(out, err).start(argv)
8
+
9
+ def initialize(out, err)
10
+ @out, @err = out, err
11
+ @options = { public: "public", state: ".index_now.json" }
12
+ end
13
+
14
+ def start(argv)
15
+ case parser.parse(argv)
16
+ in _ if @options[:help] then usage(@out, 0)
17
+ in [ "init" ] then init
18
+ in [ "pending" ] then pending
19
+ in [ "announce" ] then announce
20
+ else usage(@err, 1)
21
+ end
22
+ rescue StandardError => error
23
+ @err.puts "index_now: #{error.message}"
24
+ 1
25
+ end
26
+
27
+ private
28
+ def init
29
+ raise Error, "#{public_dir}/ is not a directory" unless Dir.exist?(public_dir)
30
+
31
+ if (existing = key_files.first)
32
+ @out.puts "#{existing} already holds the key"
33
+ else
34
+ path = File.join(public_dir, "#{valid(@options[:key] || Key.generate)}.txt")
35
+ File.write(path, File.basename(path, ".txt"))
36
+ @out.puts "wrote #{path}: deploy it with the site, then run `index_now announce`"
37
+ end
38
+ 0
39
+ end
40
+
41
+ def pending
42
+ urls = store.due(entries).map(&:url)
43
+ @out.puts(urls.empty? ? "nothing to announce" : urls)
44
+ 0
45
+ end
46
+
47
+ def announce
48
+ key = self.key
49
+ sent = @options[:all] ? entries : store.due(entries)
50
+ urls = sent.map(&:url)
51
+ Client.submit!(urls, key: key)
52
+ store.record!(sent)
53
+
54
+ if urls.empty?
55
+ @out.puts "nothing to announce"
56
+ else
57
+ @out.puts "announced #{urls.size} url#{"s" unless urls.one?}"
58
+ urls.each { |url| @out.puts " #{url}" }
59
+ end
60
+ 0
61
+ end
62
+
63
+ def entries = @entries ||= Sitemap.parse(File.read(sitemap), from: sitemap)
64
+
65
+ def store = @store ||= FileStore.new(@options[:state])
66
+
67
+ def public_dir = @options[:public]
68
+
69
+ def sitemap = @options[:sitemap] || File.join(public_dir, "sitemap.xml")
70
+
71
+ def key
72
+ return valid(@options[:key]) if @options[:key]
73
+
74
+ files = key_files
75
+ raise Error, "no key file in #{public_dir}/: run `index_now init`, or pass --key" if files.empty?
76
+ raise Error, "#{files.join(" and ")} are all key files: pass --key" if files.size > 1
77
+
78
+ File.basename(files.first, ".txt")
79
+ end
80
+
81
+ # A key file is <key>.txt holding exactly that key; other text files (robots.txt, llms.txt) are not.
82
+ def key_files
83
+ Dir[File.join(public_dir, "*.txt")].sort.select do |path|
84
+ name = File.basename(path, ".txt")
85
+ Key.valid?(name) && File.read(path).strip == name
86
+ end
87
+ end
88
+
89
+ def valid(key)
90
+ raise Error, "an IndexNow key is 8 to 128 letters, digits or dashes" unless Key.valid?(key)
91
+
92
+ key
93
+ end
94
+
95
+ def usage(io, status)
96
+ io.puts parser
97
+ status
98
+ end
99
+
100
+ def parser
101
+ @parser ||= OptionParser.new do |o|
102
+ o.banner = <<~TEXT
103
+ Usage: index_now <command> [options]
104
+
105
+ Announces a static site's changed pages to IndexNow, from its built sitemap.
106
+
107
+ Commands:
108
+ init write a key file into the public directory
109
+ pending list the urls the next announce would send
110
+ announce send them, and record what was sent
111
+
112
+ Options:
113
+ TEXT
114
+ o.on("--public DIR", "the built site (default: public)") { |v| @options[:public] = v }
115
+ o.on("--sitemap PATH", "the sitemap (default: <public>/sitemap.xml)") { |v| @options[:sitemap] = v }
116
+ o.on("--state PATH", "what was announced (default: .index_now.json)") { |v| @options[:state] = v }
117
+ o.on("--key KEY", "the IndexNow key (default: the key file in <public>)") { |v| @options[:key] = v }
118
+ o.on("--all", "announce: send every url, e.g. after rotating the key") { @options[:all] = true }
119
+ o.on("-h", "--help", "show this") { @options[:help] = true }
120
+ end
121
+ end
122
+ end
123
+ end
@@ -9,11 +9,11 @@ module IndexNow
9
9
  BATCH = 10_000
10
10
 
11
11
  class << self
12
- def submit!(urls)
12
+ def submit!(urls, key:)
13
13
  return 0 if urls.empty?
14
14
 
15
15
  host = host_of(urls)
16
- urls.each_slice(BATCH) { |batch| post(host, batch) }
16
+ urls.each_slice(BATCH) { |batch| post(host, key, batch) }
17
17
  urls.size
18
18
  end
19
19
 
@@ -26,26 +26,24 @@ module IndexNow
26
26
  host
27
27
  end
28
28
 
29
- def post(host, urls)
29
+ def post(host, key, urls)
30
30
  request = Net::HTTP::Post.new(ENDPOINT)
31
31
  request["Content-Type"] = "application/json; charset=utf-8"
32
- request.body = JSON.generate(host: host, key: key, keyLocation: key_location(host), urlList: urls)
32
+ request.body = JSON.generate(host: host, key: key, keyLocation: key_location(host, key), urlList: urls)
33
33
 
34
34
  response = Net::HTTP.start(ENDPOINT.host, ENDPOINT.port, use_ssl: true, open_timeout: 10, read_timeout: 20) do |http|
35
35
  http.request(request)
36
36
  end
37
37
  code = response.code.to_i
38
- raise SubmitFailed, failure_message(code, host) unless [ 200, 202 ].include?(code)
38
+ raise SubmitFailed, failure_message(code, host, key) unless [ 200, 202 ].include?(code)
39
39
  end
40
40
 
41
- def key = IndexNow.config.key
41
+ def key_location(host, key) = "https://#{host}/#{key}.txt"
42
42
 
43
- def key_location(host) = "https://#{host}/#{key}.txt"
44
-
45
- def failure_message(code, host)
43
+ def failure_message(code, host, key)
46
44
  case code
47
45
  when 400 then "400 malformed request: the JSON body was rejected"
48
- when 403 then "403 key not valid: #{key_location(host)} must be reachable and contain exactly the key"
46
+ when 403 then "403 key not valid: #{key_location(host, key)} must be reachable and contain exactly the key"
49
47
  when 422 then "422 the urls do not belong to #{host}, or the key does not match the schema"
50
48
  when 429 then "429 too many requests: rate limited as potential spam"
51
49
  else "IndexNow answered HTTP #{code}"
@@ -0,0 +1,37 @@
1
+ require "json"
2
+
3
+ module IndexNow
4
+ # What a static site announced, as { url => lastmod } in a JSON file; the counterpart of Announcement.
5
+ class FileStore
6
+ def initialize(path)
7
+ @path = path
8
+ end
9
+
10
+ def due(entries) = entries.select { |entry| entry.due?(announced) }
11
+
12
+ def record!(entries)
13
+ return if entries.empty?
14
+
15
+ recorded = announced.merge(entries.to_h { |entry| [ entry.url, entry.lastmod ] })
16
+ write(recorded.sort.to_h.transform_values { |lastmod| lastmod&.iso8601(lastmod.usec.zero? ? 0 : 6) })
17
+ @announced = recorded
18
+ end
19
+
20
+ private
21
+ def announced
22
+ @announced ||= File.exist?(@path) ? read : {}
23
+ end
24
+
25
+ def read
26
+ JSON.parse(File.read(@path)).transform_values { |lastmod| lastmod && Time.iso8601(lastmod) }
27
+ rescue JSON::ParserError, ArgumentError => error
28
+ raise Error, "#{@path} is not an index_now state file: #{error.message}"
29
+ end
30
+
31
+ # Through a temp file, so a crash mid-write cannot leave half a file that would re-announce everything.
32
+ def write(hash)
33
+ File.write("#{@path}.tmp", "#{JSON.pretty_generate(hash)}\n")
34
+ File.rename("#{@path}.tmp", @path)
35
+ end
36
+ end
37
+ end
@@ -0,0 +1,11 @@
1
+ require "securerandom"
2
+
3
+ module IndexNow
4
+ module Key
5
+ FORMAT = /\A[a-zA-Z0-9-]{8,128}\z/
6
+
7
+ def self.generate = SecureRandom.hex(16)
8
+
9
+ def self.valid?(key) = FORMAT.match?(key)
10
+ end
11
+ end
@@ -4,11 +4,21 @@ module IndexNow
4
4
  class SitemapError < Error; end
5
5
 
6
6
  module Sitemap
7
- Entry = Data.define(:url, :lastmod)
7
+ Entry = Data.define(:url, :lastmod) do
8
+ # announced: { url => lastmod last announced for it }
9
+ def due?(announced)
10
+ return true unless announced.key?(url)
11
+
12
+ previous = announced[url]
13
+ lastmod && (previous.nil? || lastmod > previous)
14
+ end
15
+ end
8
16
 
9
17
  class << self
10
- def entries
11
- parse(fetch).xpath("/urlset/url").map do |url|
18
+ def entries = parse(fetch, from: url)
19
+
20
+ def parse(xml, from:)
21
+ document(xml, from).xpath("/urlset/url").map do |url|
12
22
  Entry.new(url: url.at("loc").text.strip, lastmod: url.at("lastmod")&.text&.then { |text| lastmod(text) })
13
23
  end.uniq(&:url)
14
24
  end
@@ -35,14 +45,14 @@ module IndexNow
35
45
  "#{status} from #{url}#{hint}"
36
46
  end
37
47
 
38
- def parse(xml)
48
+ def document(xml, from)
39
49
  document = Nokogiri::XML(xml, &:strict).remove_namespaces!
40
50
  root = document.root&.name
41
- raise SitemapError, "#{url} is a #{root || "empty document"}, not a urlset" unless root == "urlset"
51
+ raise SitemapError, "#{from} is a #{root || "empty document"}, not a urlset" unless root == "urlset"
42
52
 
43
53
  document
44
54
  rescue Nokogiri::XML::SyntaxError => error
45
- raise SitemapError, "#{url} is not an XML urlset: #{error.message}"
55
+ raise SitemapError, "#{from} is not an XML urlset: #{error.message}"
46
56
  end
47
57
 
48
58
  def lastmod(text)
@@ -1,3 +1,3 @@
1
1
  module IndexNow
2
- VERSION = "0.1.0"
2
+ VERSION = "0.2.1"
3
3
  end
data/lib/index_now.rb CHANGED
@@ -6,10 +6,12 @@ module IndexNow
6
6
  end
7
7
 
8
8
  require "index_now/configuration"
9
- require "index_now/engine"
9
+ require "index_now/key"
10
+ require "index_now/engine" if defined?(Rails::Engine)
10
11
  require "index_now/result"
11
12
  require "index_now/sitemap"
12
13
  require "index_now/client"
14
+ require "index_now/file_store"
13
15
 
14
16
  module IndexNow
15
17
  class << self
@@ -32,7 +34,7 @@ module IndexNow
32
34
 
33
35
  entries = entries.call
34
36
  urls = entries.map(&:url)
35
- Client.submit!(urls)
37
+ Client.submit!(urls, key: config.key)
36
38
  Announcement.record!(entries)
37
39
  Rails.logger.info("IndexNow announced #{urls.join(", ")}") if urls.any?
38
40
  Result.new(urls: urls)
metadata CHANGED
@@ -1,28 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: index_now
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
4
+ version: 0.2.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Anton Kopylov
8
- bindir: bin
8
+ bindir: exe
9
9
  cert_chain: []
10
10
  date: 1980-01-02 00:00:00.000000000 Z
11
11
  dependencies:
12
- - !ruby/object:Gem::Dependency
13
- name: rails
14
- requirement: !ruby/object:Gem::Requirement
15
- requirements:
16
- - - "~>"
17
- - !ruby/object:Gem::Version
18
- version: '8.0'
19
- type: :runtime
20
- prerelease: false
21
- version_requirements: !ruby/object:Gem::Requirement
22
- requirements:
23
- - - "~>"
24
- - !ruby/object:Gem::Version
25
- version: '8.0'
26
12
  - !ruby/object:Gem::Dependency
27
13
  name: nokogiri
28
14
  requirement: !ruby/object:Gem::Requirement
@@ -37,10 +23,11 @@ dependencies:
37
23
  - - "~>"
38
24
  - !ruby/object:Gem::Version
39
25
  version: '1.16'
40
- description: Reads the app's own sitemap, announces every URL whose lastmod moved
26
+ description: Reads the site's own sitemap, announces every URL whose lastmod moved
41
27
  since it was last announced to api.indexnow.org, and records what it sent so nothing
42
- is announced twice.
43
- executables: []
28
+ is announced twice. A Rails engine for apps, an index_now command for static sites.
29
+ executables:
30
+ - index_now
44
31
  extensions: []
45
32
  extra_rdoc_files: []
46
33
  files:
@@ -48,14 +35,17 @@ files:
48
35
  - LICENSE.txt
49
36
  - README.md
50
37
  - app/models/index_now/announcement.rb
51
- - db/migrate/20261028000000_create_index_now_announcements.rb
38
+ - exe/index_now
52
39
  - lib/generators/index_now/install/install_generator.rb
53
40
  - lib/generators/index_now/install/templates/create_index_now_announcements.rb.tt
54
41
  - lib/generators/index_now/install/templates/initializer.rb.tt
55
42
  - lib/index_now.rb
43
+ - lib/index_now/cli.rb
56
44
  - lib/index_now/client.rb
57
45
  - lib/index_now/configuration.rb
58
46
  - lib/index_now/engine.rb
47
+ - lib/index_now/file_store.rb
48
+ - lib/index_now/key.rb
59
49
  - lib/index_now/result.rb
60
50
  - lib/index_now/sitemap.rb
61
51
  - lib/index_now/version.rb
@@ -64,7 +54,6 @@ homepage: https://github.com/tonic20/index_now
64
54
  licenses:
65
55
  - MIT
66
56
  metadata:
67
- homepage_uri: https://github.com/tonic20/index_now
68
57
  source_code_uri: https://github.com/tonic20/index_now
69
58
  changelog_uri: https://github.com/tonic20/index_now/blob/main/CHANGELOG.md
70
59
  rdoc_options: []
@@ -83,5 +72,5 @@ required_rubygems_version: !ruby/object:Gem::Requirement
83
72
  requirements: []
84
73
  rubygems_version: 3.7.2
85
74
  specification_version: 4
86
- summary: Announce a Rails app's changed sitemap URLs to IndexNow
75
+ summary: Announce a Rails app's or a static site's changed sitemap URLs to IndexNow
87
76
  test_files: []
@@ -1,10 +0,0 @@
1
- class CreateIndexNowAnnouncements < ActiveRecord::Migration[8.0]
2
- def change
3
- create_table :index_now_announcements do |t|
4
- t.string :url, null: false
5
- t.datetime :lastmod
6
- t.datetime :announced_at, null: false
7
- end
8
- add_index :index_now_announcements, :url, unique: true
9
- end
10
- end