indexmap 0.8.2 → 0.9.0

data/docs/storage.md

@@ -0,0 +1,94 @@
+# Storage contract
+
+> Atomic publication and streamed delivery describe unreleased main.
+> See [release status](../README.md#release-status).
+
+All outputs share `config.storage`. Operation instances resolve storage lazily
+once; construct a new runner, validator, creator, or pinger for a new storage
+context. Collaborators receive the same adapter through `storage:`. Configure
+public URLs where crawlers can fetch artifacts, even when the backing objects are private. Built-in persistent
+adapters are `Indexmap::Storage::Filesystem` and
+`Indexmap::Storage::ActiveStorage`; the latter uses a host-owned model and attachment.
+`Storage::Memory` is useful in tests and retains entire bodies in memory.
+
+## Legacy adapters
+
+The six-method contract remains supported without atomic publication:
+
+```ruby
+storage.write(filename, body, content_type: "application/xml")
+storage.read(filename)                 # XML string; missing keys should raise
+storage.exist?(filename)               # boolean
+storage.list(prefix: nil, suffix: nil) # sorted relative keys matching both filters
+storage.delete(filename)
+storage.public_url(filename)           # public URL for this key
+```
+
+`prefix` and `suffix` match the whole relative key, not just its basename.
+Write/delete return values are not used by the generation pipeline. Custom
+adapters must reject traversal outside their storage namespace and accept nested
+relative keys. Restrict the configured filesystem directory to sitemap artifacts:
+its `list` method restricts traversal to the prefix's literal directory and
+leading filename fragment. Calls without a prefix scan the whole visible tree.
+Hidden paths and symlinks are excluded; prefixes must be normalized relative
+paths without traversal, absolute paths, or redundant separators.
+
+## Streaming and metadata
+
+These optional methods avoid whole-file read/write fallbacks during generation
+and validation. Both are mandatory for the sitemap HTTP controller's reads:
+
+```ruby
+storage.open(filename) { |io| ... } # readable binary IO, closed after the block
+storage.metadata(filename)         # {etag:, bytes:, last_modified:}
+storage.write_io(filename, io, content_type: "application/xml")
+```
+
+An ETag must change whenever content changes; `bytes` is uncompressed byte length
+and `last_modified` is a time. Active Storage also provides `generation` metadata
+for active roots. `write_io` reads from the current IO position. Validation and
+parsing use `read` when `open` plus `metadata` are unavailable. Generation uses
+`write` when `write_io` is unavailable. These fallbacks retain a whole bounded
+file in memory.
+
+## Atomic publication and pruning
+
+```ruby
+storage.revision(root) # opaque revision token; nil when the root does not exist
+storage.publish(staged_storage, files: filenames, root: root, expected: revision)
+storage.prune(root: root, before: cutoff_time) # deleted relative paths
+```
+
+`files` lists staged children and the root. `staged_storage.open(key)` supplies
+validated XML. An atomic adapter must preserve immutable child keys, coordinate
+activation against `expected`, and raise `Indexmap::SupersededPublication` on a
+revision mismatch. Root replacement must expose a complete publication; preserve
+previously published children for cached roots. A failed publish must not activate
+a partial root. Built-in adapters clean up ordinary failed builds immediately;
+crashes/outages can still leave orphans. `publish` has no required return value.
+
+Atomic mode requires `revision` and `publish`; pruning additionally requires
+`prune`. The shared HTTP controller uses the built-in `Publication` ownership and
+archive naming contract. Custom atomic storage intended for that controller must
+also preserve those archives and expose `open`/metadata. Do not claim controller
+compatibility from `publish` alone.
+
+Pruning must coordinate with activation, preserve the current generation and
+other outputs, and remove only expired artifacts owned by the selected root.
+Archive retirement time controls the retained generation's grace period.
+
+Active Storage rejects stale builds before uploading, checks again before each
+upload, and destroys/purges its failed build's new records and blobs outside the
+root lock. Activation verifies child existence under that lock. Work already in
+flight can be lost to a race; crashes still need retention/unattached cleanup.
+`PublicationCleanupError#failures` identifies cleanup artifacts and errors; the
+original publication error remains its cause. Preserve object-key diagnostics if
+service deletion fails after a blob row has been destroyed.
+
+Filesystem copies and root-archive preparation occur in private destination
+staging outside the root lock. Activation uses renames under the lock. Each build
+holds its own staging lock; prune skips live builds and reclaims abandoned
+staging after the grace period, returning private directory paths too. See
+[publication and retention](dynamic-sitemaps.md#atomic-publication-and-storage)
+for filesystem semantics, the required unique Active Storage filename index,
+attachment cleanup, and retention operations.

data/lib/generators/indexmap/install_generator.rb

@@ -28,7 +28,7 @@ module Indexmap
           indexmap_tasks_requested = Rake.application.options.show_tasks ||
             ARGV.any? { |argument| ["-T", "--tasks"].include?(argument) }
           require "indexmap/tasks" if indexmap_tasks_requested ||
-            ARGV.any? { |argument| argument.start_with?("indexmap:") }
+            (Rake.application.top_level_tasks + ARGV).any? { |argument| argument.start_with?("indexmap:") }
 
         RUBY
       end

data/lib/generators/indexmap/templates/initializer.rb.tt

@@ -1,33 +1,33 @@
 # frozen_string_literal: true
 
-module IndexmapConfiguration
-  MUTEX = Mutex.new
+module Indexmap
+  module HostConfiguration
+    MUTEX = Mutex.new
 
-  def self.apply
-    return if @applied
-
-    MUTEX.synchronize do
+    def self.apply
       return if @applied
 
-      require "indexmap"
+      MUTEX.synchronize do
+        return if @applied
+
+        require "indexmap"
 
-      Indexmap.configure do |config|
-        config.base_url = -> { ENV.fetch("INDEXMAP_BASE_URL", "http://localhost:3000") }
-        config.storage = lambda {
-          Indexmap::Storage::Filesystem.new(
+        Indexmap.configure do |config|
+          config.base_url = -> { ENV.fetch("INDEXMAP_BASE_URL", "http://localhost:3000") }
+          config.storage = Indexmap::Storage::Filesystem.new(
             path: Rails.public_path,
             public_url: config.base_url
           )
-        }
 
-        # Search-engine pings start from config.index_filename (default: "sitemap.xml").
-        # Configure the sitemap data owned by your application:
-        # config.sections = -> { Sitemap.sections }
-      end
+          # Search-engine pings start from config.index_filename (default: "sitemap.xml").
+          # Configure the sitemap data owned by your application:
+          # config.sections = -> { Sitemap.sections }
+        end
 
-      @applied = true
+        @applied = true
+      end
     end
   end
 end
 
-IndexmapConfiguration.apply if defined?(Indexmap)
+Indexmap::HostConfiguration.apply if defined?(Indexmap::Configuration)

data/lib/indexmap/artifact.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+module Indexmap
+  Artifact = Struct.new(:filename, :body, :content_type) do
+    def basename
+      File.basename(filename)
+    end
+  end
+end

data/lib/indexmap/configuration.rb

@@ -4,6 +4,8 @@ module Indexmap
   class Configuration
     VALID_FORMATS = %i[index single_file].freeze
 
+    attr_accessor :atomic, :max_urls, :max_bytes
+
     attr_writer :base_url, :entries, :format, :index_filename, :sections, :storage
 
     def initialize
@@ -18,7 +20,8 @@ module Indexmap
     end
 
     def entries
-      Array(resolve(@entries))
+      value = resolve(@entries)
+      value.is_a?(Enumerable) ? value : Array(value)
     end
 
     def format
@@ -38,6 +41,10 @@ module Indexmap
       @index_now ||= IndexNowConfiguration.new
     end
 
+    def robots
+      @robots ||= Robots.new(configuration: self)
+    end
+
     def storage
       resolve(@storage) || Storage::Filesystem.new(path: "public", public_url: base_url)
     end
@@ -46,6 +53,10 @@ module Indexmap
       Array(resolve(@sections))
     end
 
+    def output_names
+      [:default, *@outputs.keys].uniq
+    end
+
     def output(name)
       output = output_for(name)
       yield(output) if block_given?

data/lib/indexmap/creator.rb

@@ -1,62 +1,65 @@
 # frozen_string_literal: true
 
-require "nokogiri"
+require "tmpdir"
+require "securerandom"
 
 module Indexmap
   class Creator
     ValidationConfiguration = Struct.new(:base_url, :index_filename, :storage)
 
-    def initialize(output:)
+    attr_reader :sitemaps
+
+    def initialize(output:, storage: nil)
       @output = output
+      @storage = storage
     end
 
     def create
-      files = format(write)
-      validate(files)
-      publish(files)
-      files.map(&:filename)
+      atomic = output.atomic
+      root = output.index_filename
+      raise ConfigurationError, "Root filename uses the reserved immutable namespace" if Publication.immutable?(root)
+      if atomic && !(storage.respond_to?(:revision) && storage.respond_to?(:publish))
+        raise ConfigurationError, "Atomic publication requires storage revision and publish capabilities"
+      end
+      expected = storage.revision(root) if atomic
+      writer = output.writer(generation: atomic ? SecureRandom.hex(16) : nil)
+      files = []
+      Dir.mktmpdir("indexmap-build") do |directory|
+        staged = Storage::Filesystem.new(path: directory)
+        writer.each_file do |file|
+          raise ConfigurationError, "Duplicate sitemap filename: #{file.filename}" if staged.exist?(file.filename)
+
+          staged.write(file.filename, file.body, content_type: file.content_type)
+          files << file.filename
+        end
+        Validator.new(configuration: ValidationConfiguration.new(
+          base_url: output.base_url, index_filename: root, storage: staged
+        )).validate!(collect: false)
+        if atomic
+          storage.publish(staged, files: files, root: root, expected: expected)
+        else
+          files.each do |filename|
+            if storage.respond_to?(:write_io)
+              staged.open(filename) { |io| storage.write_io(filename, io, content_type: "application/xml") }
+            else
+              storage.write(filename, staged.read(filename), content_type: "application/xml")
+            end
+          end
+        end
+      end
+      @sitemaps = files.map do |filename|
+        location = storage.respond_to?(:public_url) ? storage.public_url(filename) : filename
+        {filename: filename, location: location, link_count: writer.counts.fetch(filename)}
+      end
+      files
     end
 
     private
 
     attr_reader :output
 
-    def write
-      output.writer.write
-    end
-
-    def format(files)
-      files.map do |file|
-        document = Nokogiri::XML(
-          file.body,
-          nil,
-          nil,
-          Nokogiri::XML::ParseOptions::DEFAULT_XML | Nokogiri::XML::ParseOptions::NOBLANKS
-        )
-        save_options = Nokogiri::XML::Node::SaveOptions::FORMAT | Nokogiri::XML::Node::SaveOptions::AS_XML
-
-        Storage::File.new(
-          filename: file.filename,
-          body: document.to_xml(indent: 2, save_with: save_options),
-          content_type: file.content_type
-        )
-      end
-    end
-
-    def validate(files)
-      Validator.new(
-        configuration: ValidationConfiguration.new(
-          base_url: output.base_url,
-          index_filename: output.index_filename,
-          storage: Storage::Memory.new(files)
-        )
-      ).validate!
-    end
-
-    def publish(files)
-      files.each do |file|
-        output.storage.write(file.filename, file.body, content_type: file.content_type)
-      end
+    def storage
+      @storage ||= output.storage
     end
   end
 end

data/lib/indexmap/index_now_configuration.rb

@@ -5,6 +5,8 @@ module Indexmap
     DEFAULT_ENDPOINT = "https://api.indexnow.org"
     DEFAULT_MAX_URLS_PER_REQUEST = 500
 
+    attr_accessor :max_attempts, :max_wait
+
     attr_writer :dry_run, :endpoint, :key, :key_filename, :max_urls_per_request, :write_key_file
 
     def dry_run?
@@ -40,7 +42,7 @@ module Indexmap
       value = resolve(@max_urls_per_request)
       return DEFAULT_MAX_URLS_PER_REQUEST if value.nil?
 
-      value.to_i
+      Integer(value.to_s, 10)
     end
 
     private

data/lib/indexmap/output.rb

@@ -4,6 +4,8 @@ module Indexmap
   class Output
     VALID_FORMATS = %i[index single_file].freeze
 
+    attr_writer :atomic, :max_urls, :max_bytes
+
     attr_writer :base_url, :entries, :format, :index_filename, :sections
 
     def initialize(configuration:)
@@ -17,7 +19,8 @@ module Indexmap
     def entries
       resolved_entries = resolve(@entries)
 
-      Array(resolved_entries.nil? ? configuration.entries : resolved_entries)
+      value = resolved_entries.nil? ? configuration.entries : resolved_entries
+      value.is_a?(Enumerable) ? value : Array(value)
     end
 
     def format
@@ -39,7 +42,13 @@ module Indexmap
       Array(resolved_sections.nil? ? configuration.sections : resolved_sections)
     end
 
-    def writer
+    def atomic
+      @atomic.nil? ? configuration.atomic == true : @atomic == true
+    end
+
+    def writer(generation: nil)
+      base_url = self.base_url
+      format = self.format
       raise ConfigurationError, "Indexmap base_url is not configured" if base_url.to_s.strip.empty?
 
       unless VALID_FORMATS.include?(format)
@@ -47,8 +56,10 @@ module Indexmap
       end
 
       if format == :single_file
-        raise ConfigurationError, "Indexmap entries are not configured" if entries.empty?
-      elsif sections.empty?
+        entries = self.entries
+        raise ConfigurationError, "Indexmap entries are not configured" if entries.is_a?(Array) && entries.empty?
+      else
+        sections = self.sections
         raise ConfigurationError, "Indexmap sections are not configured" if sections.empty?
       end
 
@@ -57,7 +68,10 @@ module Indexmap
         format: format,
         sections: sections,
         base_url: base_url,
-        index_filename: index_filename
+        index_filename: index_filename,
+        max_urls: @max_urls || configuration.max_urls || Validator::MAX_ENTRIES,
+        max_bytes: @max_bytes || configuration.max_bytes || Validator::MAX_BYTES,
+        generation: generation
       )
     end
 

data/lib/indexmap/parser.rb

@@ -3,6 +3,8 @@
 require "net/http"
 require "nokogiri"
 require "uri"
+require "pathname"
+require "stringio"
 
 module Indexmap
   class Parser
@@ -19,8 +21,49 @@ module Indexmap
       return reset! && entries if reset
       return @entries if defined?(@entries)
 
+      @entries = each_entry.to_a
+    end
+
+    # Unlike entries, this does not cache or retain the URL inventory.
+    def each_entry
+      return enum_for(:each_entry) unless block_given?
+
       visited = Set.new
-      @entries = parse_source(@source, visited: visited)
+      pending = [@source]
+      until pending.empty?
+        source = normalize_source(pending.pop)
+        next if source.nil? || visited.include?(source)
+
+        visited.add(source)
+        children = []
+        open_source(source) do |io|
+          reader = Nokogiri::XML::Reader.from_io(io, nil, nil, Nokogiri::XML::ParseOptions::NONET)
+          root = nil
+          reader.each do |node|
+            if node.node_type == Nokogiri::XML::Reader::TYPE_DOCUMENT_TYPE
+              raise ValidationError, "Sitemap DTDs are not allowed: #{source}"
+            end
+            next unless node.node_type == Nokogiri::XML::Reader::TYPE_ELEMENT
+
+            root = node.local_name if node.depth == 0
+            next unless node.depth == 1
+
+            if root == "sitemapindex" && node.local_name == "sitemap"
+              fields = read_fields(reader)
+              child = resolve_child_sitemap(source, fields.fetch("loc").to_s.strip)
+              children << child if child
+            elsif root == "urlset" && node.local_name == "url"
+              fields = read_fields(reader)
+              loc = fields.fetch("loc").to_s.strip
+              next if loc.empty?
+
+              lastmod = fields.fetch("lastmod").to_s.strip
+              yield Entry.new(loc: loc, lastmod: lastmod.empty? ? nil : lastmod, source_sitemap: source)
+            end
+          end
+        end
+        pending.concat(children.reverse)
+      end
     end
 
     def paths(reset: false)
@@ -60,33 +103,26 @@ module Indexmap
 
     attr_reader :index_filename, :storage
 
-    def parse_source(source, visited:)
-      normalized_source = normalize_source(source)
-      return [] if normalized_source.nil? || visited.include?(normalized_source)
-
-      visited.add(normalized_source)
-      xml = read_source(normalized_source)
-      return [] if xml.to_s.strip.empty?
-
-      document = Nokogiri::XML(xml)
-      document.remove_namespaces!
-
-      if document.at_xpath("/sitemapindex")
-        document.xpath("//sitemap/loc").flat_map do |node|
-          child_source = resolve_child_sitemap(normalized_source, node.text.to_s.strip)
-          next [] if child_source.nil?
-
-          parse_source(child_source, visited: visited)
+    def read_fields(reader)
+      fields = {"loc" => nil, "lastmod" => nil}
+      return fields if reader.self_closing?
+
+      text = nil
+      while reader.read
+        type = reader.node_type
+        if type == Nokogiri::XML::Reader::TYPE_END_ELEMENT
+          break if reader.depth == 1
+
+          text = nil if reader.depth == 2
+        elsif type == Nokogiri::XML::Reader::TYPE_ELEMENT && reader.depth == 2
+          name = reader.local_name
+          text = fields.key?(name) ? (fields[name] = +"") : nil
+        elsif text && [Nokogiri::XML::Reader::TYPE_TEXT, Nokogiri::XML::Reader::TYPE_CDATA,
+          Nokogiri::XML::Reader::TYPE_WHITESPACE, Nokogiri::XML::Reader::TYPE_SIGNIFICANT_WHITESPACE].include?(type)
+          text << reader.value
         end
-      else
-        document.xpath("//url").map do |url_node|
-          loc = url_node.at_xpath("loc")&.text.to_s.strip
-          next if loc.empty?
-
-          lastmod = url_node.at_xpath("lastmod")&.text.to_s.strip
-          Entry.new(loc: loc, lastmod: lastmod.empty? ? nil : lastmod, source_sitemap: normalized_source)
-        end.compact
       end
+      fields
     end
 
     def resolve_child_sitemap(parent_source, loc)
@@ -97,7 +133,7 @@ module Indexmap
         if remote_source?(loc)
           remote_child_source(parent_uri, loc)
         else
-          URI.join(parent_uri.to_s, loc).to_s
+          remote_child_source(parent_uri, URI.join(parent_uri.to_s, loc).to_s)
         end
       elsif remote_source?(loc)
         uri = URI.parse(loc)
@@ -120,12 +156,12 @@ module Indexmap
 
     def remote_child_source(parent_uri, loc)
       child_uri = URI.parse(loc)
-      return child_uri.to_s unless @rebase_remote_children
-      return child_uri.to_s if child_uri.host == parent_uri.host && child_uri.port == parent_uri.port && child_uri.scheme == parent_uri.scheme
-
-      child_uri.scheme = parent_uri.scheme
-      child_uri.host = parent_uri.host
-      child_uri.port = parent_uri.port
+      if @rebase_remote_children
+        child_uri.scheme = parent_uri.scheme
+        child_uri.host = parent_uri.host
+        child_uri.port = parent_uri.port
+      end
+      check_remote_origin!(child_uri)
       child_uri.to_s
     end
 
@@ -155,35 +191,59 @@ module Indexmap
       false
     end
 
-    def read_source(source)
+    def open_source(source)
       if remote_source?(source)
-        fetch_remote_source(source)
+        body = fetch_remote_source(source)
+        yield StringIO.new(body) if body && !body.strip.empty?
       elsif storage.exist?(source)
-        storage.read(source)
+        if storage.respond_to?(:open) && storage.respond_to?(:metadata)
+          check_size!(storage.metadata(source).fetch(:bytes))
+          storage.open(source) { |io| yield io }
+        else
+          body = storage.read(source)
+          check_size!(body.bytesize)
+          yield StringIO.new(body) unless body.strip.empty?
+        end
+      end
+    end
+
+    def check_size!(bytes)
+      raise ValidationError, "Sitemap exceeds #{Validator::MAX_BYTES} bytes" if bytes > Validator::MAX_BYTES
+    end
+
+    def check_remote_origin!(uri)
+      origin = URI.parse(@source)
+      unless uri.is_a?(URI::HTTP) && !uri.userinfo &&
+          [uri.scheme, uri.host, uri.port] == [origin.scheme, origin.host, origin.port]
+        raise ValidationError, "Remote sitemap must use the source origin: #{uri}"
       end
     end
 
     def fetch_remote_source(source, redirects_remaining: 3)
       uri = URI.parse(source)
+      check_remote_origin!(uri)
       request = Net::HTTP::Get.new(uri)
       request["User-Agent"] = "Indexmap::Parser/1.0"
-
-      response = Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == "https", open_timeout: 10, read_timeout: 20) do |http|
-        http.request(request)
-      end
-
-      case response
-      when Net::HTTPSuccess
-        response.body
-      when Net::HTTPRedirection
-        return if redirects_remaining <= 0
-
-        location = response["location"].to_s
-        return if location.empty?
-
-        redirected = URI.join(source, location).to_s
-        fetch_remote_source(redirected, redirects_remaining: redirects_remaining - 1)
+      redirected = nil
+      body = nil
+
+      Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == "https", open_timeout: 10, read_timeout: 20) do |http|
+        http.request(request) do |response|
+          check_size!(response.content_length) if response.content_length
+          body = +"" if response.is_a?(Net::HTTPSuccess)
+          if response.is_a?(Net::HTTPRedirection)
+            location = response["location"].to_s
+            redirected = URI.join(source, location).to_s unless location.empty? || redirects_remaining <= 0
+          end
+          bytes = 0
+          response.read_body do |chunk|
+            bytes += chunk.bytesize
+            check_size!(bytes)
+            body << chunk if body
+          end
+        end
       end
+      redirected ? fetch_remote_source(redirected, redirects_remaining: redirects_remaining - 1) : body
     rescue URI::InvalidURIError
       nil
     end

data/lib/indexmap/pinger/base.rb

@@ -10,8 +10,9 @@ module Indexmap
         new(...).ping
       end
 
-      def initialize(configuration: Indexmap.configuration)
+      def initialize(configuration: Indexmap.configuration, storage: nil)
         @configuration = configuration
+        @storage = storage
       end
 
       def ping
@@ -34,6 +35,10 @@ module Indexmap
 
       attr_reader :configuration
 
+      def storage
+        @storage ||= configuration.storage
+      end
+
       def host
         configuration.base_url
       end
@@ -46,10 +51,6 @@ module Indexmap
         hostname.sub(/\Awww\./, "")
       end
 
-      def storage
-        configuration.storage
-      end
-
       def sitemap_files
         index_filename = configuration.index_filename.to_s
         return [] if index_filename.empty? || !storage.exist?(index_filename)