index_now 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 131220889a57d1c72c46f1332899034547b103c7b9594d06de1b1f06dd3937d4
+  data.tar.gz: a78d03b12c476c130d364ad1d2a91ff9cfcbe3b1928994570730157d7db6dffc
+SHA512:
+  metadata.gz: 36d2dce172e838142c225200bb726dfdd72da7b0a14e11f53576583474ebf0dbe2d627c009f11dd39178ed424792cfa91c723908b193d61fd7faf2c240714b5c
+  data.tar.gz: 96f998799a9fdf85f2af8f5aa5228f084e8d299708892756b2b4f24e7eda6e4ca55749fca322e0e70c133f79f71c6c3b34d5baede34ee30b54367f2ecec9b196

data/CHANGELOG.md

@@ -0,0 +1,8 @@
+# Changelog
+
+## [0.1.0] — 2026-10-28
+
+- First release: announces sitemap URLs whose `lastmod` moved to IndexNow,
+  records what was sent, and never raises from `IndexNow.announce_changes`.
+- `index_now:announce`, `index_now:pending` and `index_now:announce_all` tasks.
+- `index_now:install` generator for the key file, initializer and migration.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Anton Kopylov
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,101 @@
+# index_now
+
+Tells search engines which pages of a Rails app changed, through
+[IndexNow](https://www.indexnow.org), so they recrawl them without waiting.
+
+It reads the app's own sitemap, announces every URL whose `<lastmod>` moved
+since it last announced that URL, and records what it sent. Running it when
+nothing changed sends nothing, so it is safe to call as often as you like.
+
+**What it reaches:** `api.indexnow.org` shares each submission with Bing (which
+also backs ChatGPT search and Copilot), Yandex, Seznam, Naver and Yep.
+**Google is not an IndexNow participant**; nothing here affects Google.
+
+## Install
+
+```ruby
+gem "index_now"
+```
+
+```sh
+bin/rails generate index_now:install   # or --key=<existing key>
+bin/rails db:migrate
+```
+
+The generator writes:
+
+- `public/<key>.txt`: the key file the engines fetch to verify you own the host.
+  The key is not a secret; commit it.
+- `config/initializers/index_now.rb`
+- a migration for `index_now_announcements`, one row per URL announced.
+
+```ruby
+IndexNow.configure do |config|
+  config.key = "…"
+  config.host = "example.com"          # defaults to routes.default_url_options[:host]
+  config.sitemap_path = "/sitemap.xml" # the default
+  config.enabled = Rails.env.production? # the default
+end
+```
+
+The sitemap is requested in-process (`Rails.application.call`) as
+`https://<host><sitemap_path>`, so the host must pass `config.hosts`.
+
+## The rule
+
+A URL is announced when it has never been announced, or when its `<lastmod>`
+is newer than the one last announced for it. A URL with no `<lastmod>` is
+announced once and never again, so give every page you want re-announced a
+`<lastmod>`:
+
+- **Copy:** a date you bump by hand when the page's text changes. Never the
+  deploy time; that announces every page on every deploy.
+- **Data:** the time the data behind the page last changed.
+
+`<sitemapindex>` files are not supported.
+
+## Triggering it
+
+```ruby
+IndexNow.announce_changes # => #<data IndexNow::Result urls=[…], error=nil>
+```
+
+Call it after the data is committed: the sitemap is read on a separate thread
+and connection. It never raises: a failure is logged, reported through `Rails.error`, and
+returned as `result.error`. Nothing is recorded for a URL until the POST that
+carried it succeeded, so the next call retries it.
+
+**After a deploy** (for copy), from `.kamal/hooks/post-deploy`:
+
+```sh
+if [ "$KAMAL_DESTINATION" = "production" ]; then
+  bin/kamal app exec -d production --reuse --primary "bin/rails index_now:announce" || true
+fi
+```
+
+**After the app changes data** (for data pages), from the job that wrote it:
+
+```ruby
+def perform
+  Stats.rebuild!
+  IndexNow.announce_changes
+end
+```
+
+## Tasks
+
+```sh
+bin/rails index_now:pending       # what the next announce would send
+bin/rails index_now:announce      # send it; exits 1 on failure, says so when disabled
+bin/rails index_now:announce_all  # send every sitemap URL again
+```
+
+## Rotating the key
+
+Rename `public/<old key>.txt` to the new key, put the same key in its body and in
+`config.key`, deploy, check `https://<host>/<key>.txt` answers 200, then run
+`index_now:announce_all`.
+
+## License
+
+MIT

data/app/models/index_now/announcement.rb

@@ -0,0 +1,23 @@
+module IndexNow
+  class Announcement < ActiveRecord::Base
+    self.table_name = "index_now_announcements"
+
+    def self.due(entries)
+      announced = where(url: entries.map(&:url)).pluck(:url, :lastmod).to_h
+
+      entries.select do |entry|
+        next true unless announced.key?(entry.url)
+
+        previous = announced[entry.url]
+        entry.lastmod && (previous.nil? || entry.lastmod > previous)
+      end
+    end
+
+    def self.record!(entries)
+      return if entries.empty?
+
+      now = Time.current
+      upsert_all(entries.map { |entry| { url: entry.url, lastmod: entry.lastmod, announced_at: now } }, unique_by: :url)
+    end
+  end
+end

data/db/migrate/20261028000000_create_index_now_announcements.rb

@@ -0,0 +1,10 @@
+class CreateIndexNowAnnouncements < ActiveRecord::Migration[8.0]
+  def change
+    create_table :index_now_announcements do |t|
+      t.string :url, null: false
+      t.datetime :lastmod
+      t.datetime :announced_at, null: false
+    end
+    add_index :index_now_announcements, :url, unique: true
+  end
+end

data/lib/generators/index_now/install/install_generator.rb

@@ -0,0 +1,35 @@
+require "rails/generators"
+require "rails/generators/active_record"
+require "securerandom"
+
+module IndexNow
+  module Generators
+    class InstallGenerator < Rails::Generators::Base
+      include Rails::Generators::Migration
+
+      source_root File.expand_path("templates", __dir__)
+      class_option :key, type: :string, desc: "Reuse an existing IndexNow key instead of generating one"
+
+      def self.next_migration_number(dirname) = ActiveRecord::Generators::Base.next_migration_number(dirname)
+
+      def create_key_file
+        create_file "public/#{key}.txt", key
+      end
+
+      def create_initializer
+        template "initializer.rb.tt", "config/initializers/index_now.rb"
+      end
+
+      def copy_migration
+        migration_template "create_index_now_announcements.rb.tt", "db/migrate/create_index_now_announcements.rb"
+      end
+
+      private
+        def key
+          @key ||= (options[:key] || SecureRandom.hex(16)).tap do |key|
+            raise ArgumentError, "an IndexNow key is 8 to 128 letters, digits or dashes" unless key.match?(/\A[a-zA-Z0-9-]{8,128}\z/)
+          end
+        end
+    end
+  end
+end

data/lib/generators/index_now/install/templates/create_index_now_announcements.rb.tt

@@ -0,0 +1,10 @@
+class CreateIndexNowAnnouncements < ActiveRecord::Migration[<%= ActiveRecord::Migration.current_version %>]
+  def change
+    create_table :index_now_announcements do |t|
+      t.string :url, null: false
+      t.datetime :lastmod
+      t.datetime :announced_at, null: false
+    end
+    add_index :index_now_announcements, :url, unique: true
+  end
+end

data/lib/generators/index_now/install/templates/initializer.rb.tt

@@ -0,0 +1,5 @@
+IndexNow.configure do |config|
+  config.key = "<%= key %>"
+  # config.host = "example.com"
+  # config.sitemap_path = "/sitemap.xml"
+end

data/lib/index_now/client.rb

@@ -0,0 +1,56 @@
+require "net/http"
+require "json"
+
+module IndexNow
+  class SubmitFailed < Error; end
+
+  module Client
+    ENDPOINT = URI("https://api.indexnow.org/indexnow").freeze
+    BATCH = 10_000
+
+    class << self
+      def submit!(urls)
+        return 0 if urls.empty?
+
+        host = host_of(urls)
+        urls.each_slice(BATCH) { |batch| post(host, batch) }
+        urls.size
+      end
+
+      private
+        def host_of(urls)
+          host = URI(urls.first).host
+          foreign = urls.reject { |url| URI(url).host == host }
+          raise SubmitFailed, "not on #{host}, refusing to submit: #{foreign.join(", ")}" if foreign.any?
+
+          host
+        end
+
+        def post(host, urls)
+          request = Net::HTTP::Post.new(ENDPOINT)
+          request["Content-Type"] = "application/json; charset=utf-8"
+          request.body = JSON.generate(host: host, key: key, keyLocation: key_location(host), urlList: urls)
+
+          response = Net::HTTP.start(ENDPOINT.host, ENDPOINT.port, use_ssl: true, open_timeout: 10, read_timeout: 20) do |http|
+            http.request(request)
+          end
+          code = response.code.to_i
+          raise SubmitFailed, failure_message(code, host) unless [ 200, 202 ].include?(code)
+        end
+
+        def key = IndexNow.config.key
+
+        def key_location(host) = "https://#{host}/#{key}.txt"
+
+        def failure_message(code, host)
+          case code
+          when 400 then "400 malformed request: the JSON body was rejected"
+          when 403 then "403 key not valid: #{key_location(host)} must be reachable and contain exactly the key"
+          when 422 then "422 the urls do not belong to #{host}, or the key does not match the schema"
+          when 429 then "429 too many requests: rate limited as potential spam"
+          else "IndexNow answered HTTP #{code}"
+          end
+        end
+    end
+  end
+end

data/lib/index_now/configuration.rb

@@ -0,0 +1,17 @@
+module IndexNow
+  class Configuration
+    attr_accessor :key, :sitemap_path
+    attr_writer :host, :enabled
+
+    def initialize
+      @sitemap_path = "/sitemap.xml"
+    end
+
+    def host
+      @host || Rails.application.routes.default_url_options[:host] ||
+        raise(Error, "IndexNow has no host: set config.host or routes.default_url_options[:host]")
+    end
+
+    def enabled = @enabled.nil? ? Rails.env.production? : @enabled
+  end
+end

data/lib/index_now/engine.rb

@@ -0,0 +1,9 @@
+module IndexNow
+  class Engine < ::Rails::Engine
+    isolate_namespace IndexNow
+
+    rake_tasks do
+      load File.expand_path("../tasks/index_now.rake", __dir__)
+    end
+  end
+end

data/lib/index_now/result.rb

@@ -0,0 +1,9 @@
+module IndexNow
+  Result = Data.define(:urls, :error, :disabled) do
+    def initialize(urls: [], error: nil, disabled: false) = super
+
+    def ok? = error.nil?
+
+    def disabled? = disabled
+  end
+end

data/lib/index_now/sitemap.rb

@@ -0,0 +1,55 @@
+require "nokogiri"
+
+module IndexNow
+  class SitemapError < Error; end
+
+  module Sitemap
+    Entry = Data.define(:url, :lastmod)
+
+    class << self
+      def entries
+        parse(fetch).xpath("/urlset/url").map do |url|
+          Entry.new(url: url.at("loc").text.strip, lastmod: url.at("lastmod")&.text&.then { |text| lastmod(text) })
+        end.uniq(&:url)
+      end
+
+      private
+        def url = "https://#{IndexNow.config.host}#{IndexNow.config.sitemap_path}"
+
+        # Own thread: Rails.application.call runs the executor, which would reset the caller's Current and query cache.
+        def fetch = Thread.new { request }.value
+
+        def request
+          status, _headers, body = Rails.application.call(Rack::MockRequest.env_for(url, "HTTP_HOST" => IndexNow.config.host))
+          content = +""
+          body.each { |chunk| content << chunk }
+          raise SitemapError, failure_message(status) unless status == 200
+
+          content
+        ensure
+          body.close if body.respond_to?(:close)
+        end
+
+        def failure_message(status)
+          hint = " (is #{IndexNow.config.host} in config.hosts?)" if status == 403
+          "#{status} from #{url}#{hint}"
+        end
+
+        def parse(xml)
+          document = Nokogiri::XML(xml, &:strict).remove_namespaces!
+          root = document.root&.name
+          raise SitemapError, "#{url} is a #{root || "empty document"}, not a urlset" unless root == "urlset"
+
+          document
+        rescue Nokogiri::XML::SyntaxError => error
+          raise SitemapError, "#{url} is not an XML urlset: #{error.message}"
+        end
+
+        def lastmod(text)
+          text = text.strip
+          text = "#{text}T00:00:00Z" if text.match?(/\A\d{4}-\d{2}-\d{2}\z/)
+          Time.iso8601(text.sub(/T(\d{2}:\d{2})(?=Z|[+-])/, 'T\1:00')).utc.floor(6)
+        end
+    end
+  end
+end

data/lib/index_now/version.rb

@@ -0,0 +1,3 @@
+module IndexNow
+  VERSION = "0.1.0"
+end

data/lib/index_now.rb

@@ -0,0 +1,55 @@
+require "time"
+require "index_now/version"
+
+module IndexNow
+  class Error < StandardError; end
+end
+
+require "index_now/configuration"
+require "index_now/engine"
+require "index_now/result"
+require "index_now/sitemap"
+require "index_now/client"
+
+module IndexNow
+  class << self
+    def config = @config ||= Configuration.new
+
+    def configure = yield(config)
+
+    def reset_config! = @config = nil
+
+    def pending = Announcement.due(Sitemap.entries)
+
+    def announce_changes = announce { pending }
+
+    def announce_all = announce { Sitemap.entries }
+
+    private
+      # Never raises: a failed announcement must not fail the deploy or job that asked for it.
+      def announce(&entries)
+        return dry_run(&entries) unless config.enabled
+
+        entries = entries.call
+        urls = entries.map(&:url)
+        Client.submit!(urls)
+        Announcement.record!(entries)
+        Rails.logger.info("IndexNow announced #{urls.join(", ")}") if urls.any?
+        Result.new(urls: urls)
+      rescue StandardError => error
+        Rails.logger.error("IndexNow failed: #{error.class}: #{error.message}")
+        Rails.error.report(error, handled: true, source: "index_now")
+        Result.new(error: error)
+      end
+
+      # Staging and development often cannot read their own sitemap (basic auth, config.hosts); that is not an error there.
+      def dry_run
+        urls = yield.map(&:url)
+        Rails.logger.info("IndexNow disabled; would announce #{urls.join(", ")}") if urls.any?
+        Result.new(disabled: true)
+      rescue StandardError => error
+        Rails.logger.info("IndexNow disabled; could not read the sitemap: #{error.message}")
+        Result.new(disabled: true)
+      end
+  end
+end

data/lib/tasks/index_now.rake

@@ -0,0 +1,30 @@
+namespace :index_now do
+  report = lambda do |result|
+    abort "IndexNow failed: #{result.error.message}" unless result.ok?
+
+    if result.disabled?
+      puts "IndexNow is disabled in #{Rails.env}; nothing sent"
+    elsif result.urls.empty?
+      puts "nothing to announce"
+    else
+      puts "announced #{result.urls.size} url#{"s" unless result.urls.one?}"
+      result.urls.each { |url| puts "  #{url}" }
+    end
+  end
+
+  desc "Announce sitemap urls whose lastmod moved since they were last announced"
+  task announce: :environment do
+    report.call(IndexNow.announce_changes)
+  end
+
+  desc "Announce every sitemap url again, e.g. after rotating the key"
+  task announce_all: :environment do
+    report.call(IndexNow.announce_all)
+  end
+
+  desc "List the urls the next announce would send"
+  task pending: :environment do
+    urls = IndexNow.pending.map(&:url)
+    puts urls.empty? ? "nothing to announce" : urls
+  end
+end

metadata

@@ -0,0 +1,87 @@
+--- !ruby/object:Gem::Specification
+name: index_now
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Anton Kopylov
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '8.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '8.0'
+- !ruby/object:Gem::Dependency
+  name: nokogiri
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.16'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.16'
+description: Reads the app's own sitemap, announces every URL whose lastmod moved
+  since it was last announced to api.indexnow.org, and records what it sent so nothing
+  is announced twice.
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE.txt
+- README.md
+- app/models/index_now/announcement.rb
+- db/migrate/20261028000000_create_index_now_announcements.rb
+- lib/generators/index_now/install/install_generator.rb
+- lib/generators/index_now/install/templates/create_index_now_announcements.rb.tt
+- lib/generators/index_now/install/templates/initializer.rb.tt
+- lib/index_now.rb
+- lib/index_now/client.rb
+- lib/index_now/configuration.rb
+- lib/index_now/engine.rb
+- lib/index_now/result.rb
+- lib/index_now/sitemap.rb
+- lib/index_now/version.rb
+- lib/tasks/index_now.rake
+homepage: https://github.com/tonic20/index_now
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/tonic20/index_now
+  source_code_uri: https://github.com/tonic20/index_now
+  changelog_uri: https://github.com/tonic20/index_now/blob/main/CHANGELOG.md
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.3'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.7.2
+specification_version: 4
+summary: Announce a Rails app's changed sitemap URLs to IndexNow
+test_files: []