indexmap 0.5.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 38fa952c31358e79a900d348041a2475d5f597d19652837b50d27713f2004250
-  data.tar.gz: 6889768f6be1e01b6de1938687a757b050a52f7f9dbf2af58d1818cca2fdc977
+  metadata.gz: 90b723b27367659107827ca3ff1f7473c6a4a917b62ae1cdb4ae2cfcf440f3f2
+  data.tar.gz: cd5be4391a83b1378efcad593408648822dabff4dc8e5479e6ae3e3c12d93ac3
 SHA512:
-  metadata.gz: 612858ebdac07d01107af653411182685e161b1e501670e569d84265e1dbcd455dbb6a7e55615f802d112037875d9eb15d94c11747a2c15e6cbd1b9f63b804de
-  data.tar.gz: 512df6e55dbad711516e9558cda07b0b90b65f7f950441e36699fe696965296690c52a478247b707ff9e15801b49b73a5d3428cea92dd1e0b49e9861c14580d2
+  metadata.gz: '099bf5472fae8a8be3d17ad40003ea5802dd410799431af367e604dca3cd0fce04e98555526dfb36b98f270774b41eec160a47c5abdc9fd81ac148080901d96f'
+  data.tar.gz: e53e45fe051e02b0000e05c5000f8f23c2e1cc7ab78a952cad6a4a33891b108c8811cd67d9fcadea6cf39bea4abf74eae9736d9e85402e6a1988ef49ace9c0dc

data/CHANGELOG.md

@@ -5,12 +5,12 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [0.5.0] - 2026-04-24
+## [0.7.0] - 2026-05-08
 
 
 ### Fixed
 
-- namespace rake tasks and harden sitemap validation (#8)
+- avoid rewriting existing IndexNow key files (#11)
 
 
 

data/README.md

@@ -34,6 +34,10 @@ Or install it directly:
 gem install indexmap
 ```
 
+Upgrading an existing app? Read [UPGRADE.md](UPGRADE.md) before deploying,
+especially if the app uses custom storage or stores sitemap files under a
+directory prefix such as `sitemaps/`.
+
 ## Ruby Usage
 
 ```ruby
@@ -51,7 +55,6 @@ sections = [
 
 Indexmap::Writer.new(
   sections: sections,
-  public_path: Pathname("public"),
   base_url: "https://example.com"
 ).write
 ```
@@ -63,7 +66,12 @@ In an initializer:
 ```ruby
 Indexmap.configure do |config|
   config.base_url = -> { "https://example.com" }
-  config.public_path = -> { Rails.public_path }
+  config.storage = -> do
+    Indexmap::Storage::Filesystem.new(
+      path: Rails.public_path,
+      public_url: config.base_url
+    )
+  end
   config.sections = -> do
     [
       Indexmap::Section.new(
@@ -85,23 +93,26 @@ bin/rails indexmap:sitemap:format
 bin/rails indexmap:sitemap:validate
 ```
 
-`indexmap:sitemap:create` is the main task. It writes sitemap files, formats them, and validates the result.
+`indexmap:sitemap:create` is the main task. It builds sitemap files in memory,
+formats them, validates the result, then writes the final XML files to the
+configured storage. Existing sitemap files are left untouched if generation or
+validation fails.
 
 ### Default Index Mode
 
 This is the default behavior. `indexmap` writes:
 
-- `public/sitemap.xml` as a sitemap index
+- `sitemap.xml` as a sitemap index
 - one or more child sitemap files from `config.sections`
 
 ### Single-File Mode
 
-For sites that only want one `public/sitemap.xml` file:
+For sites that only want one `sitemap.xml` file:
 
 ```ruby
 Indexmap.configure do |config|
   config.base_url = -> { "https://example.com" }
-  config.public_path = -> { Rails.public_path }
+  config.storage = -> { Indexmap::Storage::Filesystem.new(path: Rails.public_path, public_url: config.base_url) }
   config.format = :single_file
   config.entries = -> do
     [
@@ -114,12 +125,135 @@ end
 
 In `:single_file` mode, `indexmap` writes a `urlset` directly to `sitemap.xml` and reads entries from `config.entries` instead of `config.sections`.
 
+### Named Outputs
+
+Most apps only need the default output. Use named outputs when one part of the
+sitemap must be generated separately, for example when static pages can be
+generated during deploy but database-heavy pages should refresh later. Named
+outputs write through the same configured storage as the default output.
+
+```ruby
+Indexmap.configure do |config|
+  config.base_url = -> { "https://example.com" }
+  config.storage = -> { Indexmap::Storage::Filesystem.new(path: Rails.root.join("storage/sitemaps"), public_url: config.base_url) }
+  config.sections = -> { Sitemap.sections }
+
+  config.output :insights_data do |output|
+    output.format = :single_file
+    output.index_filename = "sitemap-insights-data.xml"
+    output.entries = -> { Sitemap.insights_data_entries }
+  end
+end
+```
+
+Generate the default output:
+
+```ruby
+Indexmap.create
+```
+
+Generate only the named output:
+
+```ruby
+Indexmap.create(:insights_data)
+```
+
+Named outputs inherit `base_url` and `format` from the main configuration unless
+you override them. Storage is configured once and shared by every output.
+
+`Indexmap.create` uses the same safe publish flow as the rake task: build,
+format, validate, and then write the final XML file or files to storage.
+
+### Storage
+
+Every `indexmap` operation reads and writes through `config.storage`. The storage
+object is the source of truth for generation, validation, parsing, Google
+submission, IndexNow submission, and IndexNow verification files.
+
+The filesystem adapter stores files in a directory and exposes public URLs from
+the same filenames:
+
+```ruby
+Indexmap.configure do |config|
+  config.base_url = "https://example.com"
+  config.storage = Indexmap::Storage::Filesystem.new(
+    path: Rails.public_path,
+    public_url: "https://example.com"
+  )
+end
+```
+
+Rails apps that store sitemap files in Active Storage can use the optional
+adapter. `indexmap` does not depend on `activestorage`; this adapter only uses
+the model and attachment object you pass in.
+
+```ruby
+Indexmap.configure do |config|
+  config.base_url = "https://example.com"
+  config.storage = Indexmap::Storage::ActiveStorage.new(
+    model: SitemapArtifact,
+    filename_column: :filename,
+    attachment: :file,
+    public_url: "https://example.com"
+  )
+end
+```
+
+Custom storage backends can implement the same small interface:
+
+```ruby
+storage.write(filename, body, content_type:)
+storage.read(filename)
+storage.exist?(filename)
+storage.list(prefix:, suffix:)
+storage.delete(filename)
+storage.public_url(filename)
+```
+
+### Deferred Dynamic Sections
+
+Use `after_create` when `indexmap:sitemap:create` should publish the default
+sitemap first, then schedule slower dynamic sections for the background. The
+callback runs only after the generated files have been formatted, validated, and
+replaced successfully.
+
+```ruby
+Indexmap.configure do |config|
+  config.base_url = -> { "https://example.com" }
+  config.storage = -> { Indexmap::Storage::Filesystem.new(path: Rails.root.join("storage/sitemaps"), public_url: config.base_url) }
+  config.sections = -> { Sitemap.sections }
+
+  config.output :insights_data do |output|
+    output.format = :single_file
+    output.index_filename = "sitemap-insights-data.xml"
+    output.entries = -> { Sitemap.insights_data_entries }
+  end
+
+  config.after_create do
+    Insights::SitemapRefreshJob.perform_later
+  end
+end
+```
+
+Then the job can stay small:
+
+```ruby
+class Insights::SitemapRefreshJob < ApplicationJob
+  def perform
+    Indexmap.create(:insights_data)
+  end
+end
+```
+
+This keeps deploys fast: the deploy only waits for `indexmap:sitemap:create`,
+while database-dependent output is refreshed by the job backend.
+
 ## Validation And Parsing
 
 `indexmap` also includes small utilities for working with generated sitemap files:
 
 ```ruby
-parser = Indexmap::Parser.new(path: Rails.public_path.join("sitemap.xml"))
+parser = Indexmap::Parser.new(source: "sitemap.xml")
 parser.paths
 # => ["/", "/about", "/articles/example"]
 
@@ -181,7 +315,7 @@ If `config.google.property` is not set, `indexmap` defaults to `sc-domain:<host>
 IndexNow submission requires a key. `indexmap` supports two ways to provide it:
 
 - set `config.index_now.key`
-- or keep a valid verification file at `public/<key>.txt`
+- or keep a valid verification file in the configured storage as `<key>.txt`
 
 Configured-key example:
 
@@ -191,7 +325,25 @@ Indexmap.configure do |config|
 end
 ```
 
-If `config.index_now.key` is set, `indexmap:sitemap:create` also writes the matching `public/<key>.txt` verification file automatically.
+If `config.index_now.key` is set, `indexmap:sitemap:create` also ensures the matching `<key>.txt` verification file exists in storage. It leaves an existing valid key file unchanged.
+
+If you need a non-standard verification filename, configure it explicitly:
+
+```ruby
+Indexmap.configure do |config|
+  config.index_now.key = -> { ENV["INDEXNOW_KEY"] }
+  config.index_now.key_filename = -> { "#{ENV.fetch("INDEXNOW_KEY")}.txt" }
+end
+```
+
+You can also disable automatic key-file writes entirely:
+
+```ruby
+Indexmap.configure do |config|
+  config.index_now.key = -> { ENV["INDEXNOW_KEY"] }
+  config.index_now.write_key_file = false
+end
+```
 
 If you prefer the file-based flow, run:
 
@@ -202,7 +354,7 @@ bin/rails indexmap:index_now:write_key
 That task:
 
 - reuses an existing valid key file when present
-- otherwise generates a new key in `public/<key>.txt`
+- otherwise generates a new key in `<key>.txt`
 - makes that key available to `indexmap:index_now:ping` without adding `config.index_now.key`
 
 If neither a configured key nor a valid key file is present, `indexmap:index_now:ping` skips IndexNow submission.

data/lib/indexmap/configuration.rb

@@ -4,11 +4,13 @@ module Indexmap
   class Configuration
     VALID_FORMATS = %i[index single_file].freeze
 
-    attr_writer :base_url, :entries, :format, :index_filename, :public_path, :sections
+    attr_writer :base_url, :entries, :format, :index_filename, :sections, :storage
 
     def initialize
       @format = :index
       @index_filename = "sitemap.xml"
+      @after_create_callbacks = []
+      @outputs = {}
     end
 
     def base_url
@@ -36,38 +38,37 @@ module Indexmap
       @index_now ||= IndexNowConfiguration.new
     end
 
-    def public_path
-      value = resolve(@public_path)
-      return Pathname("public") if value.nil?
-
-      Pathname(value)
+    def storage
+      resolve(@storage) || Storage::Filesystem.new(path: "public", public_url: base_url)
     end
 
     def sections
       Array(resolve(@sections))
     end
 
+    def output(name)
+      output = output_for(name)
+      yield(output) if block_given?
+      output
+    end
+
+    def output_for(name = :default)
+      normalized_name = name.to_sym
+      @outputs[normalized_name] ||= Output.new(configuration: self)
+    end
+
+    def after_create(&block)
+      raise ArgumentError, "after_create requires a block" unless block
+
+      @after_create_callbacks << block
+    end
+
+    def run_after_create_callbacks
+      @after_create_callbacks.each(&:call)
+    end
+
     def writer
-      raise ConfigurationError, "Indexmap base_url is not configured" if base_url.to_s.strip.empty?
-
-      unless VALID_FORMATS.include?(format)
-        raise ConfigurationError, "Indexmap format must be one of: #{VALID_FORMATS.join(", ")}"
-      end
-
-      if format == :single_file
-        raise ConfigurationError, "Indexmap entries are not configured" if entries.empty?
-      elsif sections.empty?
-        raise ConfigurationError, "Indexmap sections are not configured" if sections.empty?
-      end
-
-      Writer.new(
-        entries: entries,
-        format: format,
-        sections: sections,
-        public_path: public_path,
-        base_url: base_url,
-        index_filename: index_filename
-      )
+      output_for(:default).writer
     end
 
     private

data/lib/indexmap/creator.rb

@@ -0,0 +1,62 @@
+# frozen_string_literal: true
+
+require "nokogiri"
+
+module Indexmap
+  class Creator
+    ValidationConfiguration = Struct.new(:base_url, :index_filename, :storage, keyword_init: true)
+
+    def initialize(output:)
+      @output = output
+    end
+
+    def create
+      files = format(write)
+      validate(files)
+      publish(files)
+      files.map(&:filename)
+    end
+
+    private
+
+    attr_reader :output
+
+    def write
+      output.writer.write
+    end
+
+    def format(files)
+      files.map do |file|
+        document = Nokogiri::XML(
+          file.body,
+          nil,
+          nil,
+          Nokogiri::XML::ParseOptions::DEFAULT_XML | Nokogiri::XML::ParseOptions::NOBLANKS
+        )
+        save_options = Nokogiri::XML::Node::SaveOptions::FORMAT | Nokogiri::XML::Node::SaveOptions::AS_XML
+
+        Storage::File.new(
+          filename: file.filename,
+          body: document.to_xml(indent: 2, save_with: save_options),
+          content_type: file.content_type
+        )
+      end
+    end
+
+    def validate(files)
+      Validator.new(
+        configuration: ValidationConfiguration.new(
+          base_url: output.base_url,
+          index_filename: output.index_filename,
+          storage: Storage::Memory.new(files)
+        )
+      ).validate!
+    end
+
+    def publish(files)
+      files.each do |file|
+        output.storage.write(file.filename, file.body, content_type: file.content_type)
+      end
+    end
+  end
+end

data/lib/indexmap/index_now_configuration.rb

@@ -5,7 +5,7 @@ module Indexmap
     DEFAULT_ENDPOINT = "https://api.indexnow.org"
     DEFAULT_MAX_URLS_PER_REQUEST = 500
 
-    attr_writer :dry_run, :endpoint, :key, :key_path, :max_urls_per_request
+    attr_writer :dry_run, :endpoint, :key, :key_filename, :max_urls_per_request, :write_key_file
 
     def dry_run?
       value = resolve(@dry_run)
@@ -21,12 +21,19 @@ module Indexmap
       resolve(@key)
     end
 
-    def key_path(public_path:, key: self.key)
-      configured_path = resolve(@key_path)
-      return Pathname(configured_path) unless configured_path.to_s.strip.empty?
+    def write_key_file?
+      value = resolve(@write_key_file)
+      return !key.to_s.strip.empty? if value.nil?
+
+      value == true || value.to_s == "1"
+    end
+
+    def key_filename(key: self.key)
+      configured_filename = resolve(@key_filename)
+      return configured_filename unless configured_filename.to_s.strip.empty?
       return if key.to_s.strip.empty?
 
-      Pathname(public_path).join("#{key}.txt")
+      "#{key}.txt"
     end
 
     def max_urls_per_request

data/lib/indexmap/output.rb

@@ -0,0 +1,72 @@
+# frozen_string_literal: true
+
+module Indexmap
+  class Output
+    VALID_FORMATS = %i[index single_file].freeze
+
+    attr_writer :base_url, :entries, :format, :index_filename, :sections
+
+    def initialize(configuration:)
+      @configuration = configuration
+    end
+
+    def base_url
+      resolve(@base_url) || configuration.base_url
+    end
+
+    def entries
+      resolved_entries = resolve(@entries)
+
+      Array(resolved_entries.nil? ? configuration.entries : resolved_entries)
+    end
+
+    def format
+      value = resolve(@format) || configuration.format
+      value.nil? ? :index : value.to_sym
+    end
+
+    def index_filename
+      resolve(@index_filename) || configuration.index_filename
+    end
+
+    def storage
+      configuration.storage
+    end
+
+    def sections
+      resolved_sections = resolve(@sections)
+
+      Array(resolved_sections.nil? ? configuration.sections : resolved_sections)
+    end
+
+    def writer
+      raise ConfigurationError, "Indexmap base_url is not configured" if base_url.to_s.strip.empty?
+
+      unless VALID_FORMATS.include?(format)
+        raise ConfigurationError, "Indexmap format must be one of: #{VALID_FORMATS.join(", ")}"
+      end
+
+      if format == :single_file
+        raise ConfigurationError, "Indexmap entries are not configured" if entries.empty?
+      elsif sections.empty?
+        raise ConfigurationError, "Indexmap sections are not configured" if sections.empty?
+      end
+
+      Writer.new(
+        entries: entries,
+        format: format,
+        sections: sections,
+        base_url: base_url,
+        index_filename: index_filename
+      )
+    end
+
+    private
+
+    attr_reader :configuration
+
+    def resolve(value)
+      value.respond_to?(:call) ? value.call : value
+    end
+  end
+end

data/lib/indexmap/parser.rb

@@ -2,18 +2,17 @@
 
 require "net/http"
 require "nokogiri"
-require "pathname"
 require "uri"
 
 module Indexmap
   class Parser
     Entry = Struct.new(:loc, :lastmod, :source_sitemap, keyword_init: true)
 
-    def initialize(path: default_path, rebase_remote_children: false, index_filename: Indexmap.configuration.index_filename, public_path: Indexmap.configuration.public_path)
-      @source = path.to_s
+    def initialize(source: nil, rebase_remote_children: false, index_filename: Indexmap.configuration.index_filename, storage: Indexmap.configuration.storage)
+      @source = (source || index_filename).to_s
       @rebase_remote_children = rebase_remote_children
       @index_filename = index_filename
-      @public_path = public_path
+      @storage = storage
     end
 
     def entries(reset: false)
@@ -59,11 +58,7 @@ module Indexmap
 
     private
 
-    attr_reader :index_filename, :public_path
-
-    def default_path
-      Indexmap::Path.existing_public_path(public_path: public_path, index_filename: index_filename)
-    end
+    attr_reader :index_filename, :storage
 
     def parse_source(source, visited:)
       normalized_source = normalize_source(source)
@@ -106,12 +101,21 @@ module Indexmap
         end
       elsif remote_source?(loc)
         uri = URI.parse(loc)
-        File.join(File.dirname(parent_source), File.basename(uri.path))
+        normalize_local_source(uri.path)
       else
-        File.expand_path(loc, File.dirname(parent_source))
+        resolve_local_child_sitemap(parent_source, loc)
       end
     rescue URI::InvalidURIError
-      File.expand_path(loc, File.dirname(parent_source))
+      resolve_local_child_sitemap(parent_source, loc)
+    end
+
+    def resolve_local_child_sitemap(parent_source, loc)
+      if loc.start_with?("/")
+        normalize_local_source(loc)
+      else
+        parent_directory = File.dirname(parent_source)
+        normalize_local_source((parent_directory == ".") ? loc : File.join(parent_directory, loc))
+      end
     end
 
     def remote_child_source(parent_uri, loc)
@@ -131,12 +135,19 @@ module Indexmap
       if remote_source?(source)
         URI.parse(source).to_s
       else
-        Pathname(source).expand_path.to_s
+        normalize_local_source(source)
       end
     rescue URI::InvalidURIError
       nil
     end
 
+    def normalize_local_source(source)
+      normalized = Pathname(source.to_s).cleanpath.to_s.sub(%r{\A/+}, "")
+      return if normalized.empty? || normalized == ".." || normalized.start_with?("../")
+
+      normalized
+    end
+
     def remote_source?(value)
       uri = URI.parse(value.to_s)
       uri.is_a?(URI::HTTP) || uri.is_a?(URI::HTTPS)
@@ -147,8 +158,8 @@ module Indexmap
     def read_source(source)
       if remote_source?(source)
         fetch_remote_source(source)
-      elsif File.exist?(source)
-        File.read(source, encoding: "UTF-8")
+      elsif storage.exist?(source)
+        storage.read(source)
       end
     end
 

data/lib/indexmap/pinger/base.rb

@@ -46,8 +46,12 @@ module Indexmap
         hostname.sub(/\Awww\./, "")
       end
 
+      def storage
+        configuration.storage
+      end
+
       def sitemap_files
-        Dir.glob(configuration.public_path.join("sitemap*.xml")).sort
+        storage.list(prefix: "sitemap", suffix: ".xml")
       end
 
       def ping_sitemap(_sitemap_file)

data/lib/indexmap/pinger/google.rb

@@ -20,10 +20,11 @@ module Indexmap
           return {status: :skipped, reason: :missing_credentials}
         end
 
-        results = sitemap_files.map { |sitemap_file| ping_sitemap(sitemap_file) }
+        files = sitemap_files
+        results = files.map { |sitemap_file| ping_sitemap(sitemap_file) }
         return {status: :skipped, reason: :no_sitemaps} if results.empty?
 
-        summarize_results(results)
+        summarize_results(results, url_count: sitemap_url_count(files))
       end
 
       private
@@ -35,7 +36,7 @@ module Indexmap
       end
 
       def ping_sitemap(sitemap_file)
-        sitemap_url = URI.join(host, File.basename(sitemap_file)).to_s
+        sitemap_url = storage.public_url(sitemap_file)
 
         unless authorized?
           logger.debug("Google Search Console does not have access to the site: #{root_domain}")
@@ -90,16 +91,26 @@ module Indexmap
         )
       end
 
-      def summarize_results(results)
+      def sitemap_url_count(files)
+        files.each_with_object(Set.new) do |sitemap_file, urls|
+          Parser.new(source: sitemap_file, storage: storage).entries.each do |entry|
+            loc = entry.loc.to_s.strip
+            urls.add(loc) unless loc.empty?
+          end
+        end.count
+      end
+
+      def summarize_results(results, url_count:)
         submitted = results.select { |result| result[:status] == :submitted }
         failures = results.select { |result| result[:status] == :failed }
 
-        return {status: :submitted, sitemap_count: submitted.count, submitted: submitted} if failures.empty?
-        return {status: :failed, sitemap_count: 0, failures: failures} if submitted.empty?
+        return {status: :submitted, sitemap_count: submitted.count, url_count: url_count, submitted: submitted} if failures.empty?
+        return {status: :failed, sitemap_count: 0, url_count: 0, failures: failures} if submitted.empty?
 
         {
           status: :partial,
           sitemap_count: submitted.count,
+          url_count: url_count,
           submitted: submitted,
           failures: failures
         }