site_maps 0.0.1.beta3 → 0.1.1

data/docs/getting-started.md

@@ -0,0 +1,138 @@
+# Getting Started
+
+## Install
+
+```ruby
+# Gemfile
+gem 'site_maps'
+```
+
+```bash
+bundle install
+```
+
+## Your first sitemap
+
+Create `config/sitemap.rb`:
+
+```ruby
+SiteMaps.use(:file_system) do
+  configure do |config|
+    config.url       = 'https://example.com/sitemap.xml'
+    config.directory = File.expand_path('public', __dir__)
+  end
+
+  process do |s|
+    s.add('/',       priority: 1.0, changefreq: 'daily')
+    s.add('/about',  priority: 0.8, lastmod: Time.now)
+    s.add('/contact', priority: 0.5)
+  end
+end
+```
+
+Generate:
+
+```bash
+bundle exec site_maps generate --config-file config/sitemap.rb
+```
+
+Output: `public/sitemap.xml`.
+
+## Dynamic URLs
+
+Yield `s.add` for every URL you want indexed. Database records work naturally:
+
+```ruby
+process :posts do |s|
+  Post.published.find_each do |post|
+    s.add("/posts/#{post.slug}", lastmod: post.updated_at, priority: 0.7)
+  end
+end
+```
+
+When the URL count of a single process exceeds `max_links` (default 50,000), the file is split into `sitemap1.xml`, `sitemap2.xml`, … and a sitemap index is written at `config.url`.
+
+## Named processes
+
+Named processes get their own file and run in parallel:
+
+```ruby
+SiteMaps.use(:file_system) do
+  configure { |c| c.url = 'https://example.com/sitemap.xml'; c.directory = 'public' }
+
+  process :static do |s|
+    s.add('/')
+    s.add('/about')
+  end
+
+  process :posts, 'posts/sitemap.xml' do |s|
+    Post.find_each { |p| s.add("/posts/#{p.slug}") }
+  end
+
+  process :products, 'products/sitemap.xml' do |s|
+    Product.find_each { |p| s.add("/products/#{p.id}") }
+  end
+end
+```
+
+Run all:
+
+```bash
+bundle exec site_maps generate --config-file config/sitemap.rb --max-threads 4
+```
+
+Run one:
+
+```bash
+bundle exec site_maps generate posts --config-file config/sitemap.rb
+```
+
+See [processes.md](processes.md) for the full process DSL including parameterized templates.
+
+## Using it in Rails
+
+Add `site_maps` to your Gemfile and generate from a Rake task, a scheduled job, or your deploy pipeline. The Railtie injects URL helpers:
+
+```ruby
+# config/sitemap.rb
+SiteMaps.use(:file_system) do
+  configure do |config|
+    config.url       = 'https://example.com/sitemap.xml'
+    config.directory = Rails.public_path.to_s
+  end
+
+  process do |s|
+    s.add(s.route.root_path, priority: 1.0)
+    s.add(s.route.about_path)
+    Post.find_each { |post| s.add(s.route.post_path(post), lastmod: post.updated_at) }
+  end
+end
+```
+
+See [rails.md](rails.md) for the full Rails integration, including asset precompile hooks and the Rack middleware for serving generated sitemaps.
+
+## Uploading to S3
+
+Swap the adapter line:
+
+```ruby
+SiteMaps.use(:aws_sdk) do
+  configure do |config|
+    config.url    = 'https://my-bucket.s3.amazonaws.com/sitemap.xml'
+    config.bucket = 'my-bucket'
+    config.region = ENV['AWS_REGION']
+    # access_key_id / secret_access_key default to ENV vars
+  end
+
+  process { |s| ... }
+end
+```
+
+See [adapters.md](adapters.md) for adapter specifics and how to build your own.
+
+## Next steps
+
+- [Processes](processes.md) — split your sitemap into static and dynamic shards
+- [SEO extensions](extensions.md) — image, video, news, hreflang
+- [CLI](cli.md) — automation-friendly generate command
+- [Rack middleware](middleware.md) — serve the generated files with correct headers

data/docs/middleware.md

@@ -0,0 +1,85 @@
+# Rack Middleware
+
+`SiteMaps::Middleware` serves generated sitemap files directly from the app. Useful when you've generated to `public/sitemaps/` (filesystem adapter) and want proper `Content-Type`, gzip handling, and XSL stylesheet routing without editing your web-server config.
+
+## Basic usage
+
+```ruby
+# config/application.rb (Rails)
+config.middleware.use SiteMaps::Middleware, adapter: -> { SiteMaps.current_adapter }
+```
+
+Or inline in `config.ru`:
+
+```ruby
+require 'site_maps'
+
+use SiteMaps::Middleware, adapter: SiteMaps.current_adapter
+run MyApp
+```
+
+## Options
+
+```ruby
+use SiteMaps::Middleware,
+  adapter:        SiteMaps.current_adapter,
+  public_prefix:  nil,
+  storage_prefix: nil,
+  x_robots_tag:   'noindex, follow',
+  cache_control:  'public, max-age=3600'
+```
+
+| Option | Purpose |
+|--------|---------|
+| `adapter` | Adapter instance (or a callable returning one — useful if the adapter is reconfigured at boot). |
+| `public_prefix` | Strip from request path before lookup — e.g. `/sitemap` if your app mounts them under a sub-path. |
+| `storage_prefix` | Prepend to the lookup key — e.g. `tenants/acme` for multi-tenant layouts. |
+| `x_robots_tag` | `X-Robots-Tag` header added to served files. |
+| `cache_control` | `Cache-Control` header. |
+
+## Behavior
+
+The middleware intercepts requests for `*.xml` and `*.xml.gz` files:
+
+- Matches → serve from the adapter with `Content-Type: application/xml`, plus `X-Robots-Tag` and `Cache-Control`.
+- Gzipped sources → auto-decompress on serve so XSL stylesheets render in the browser. Clients asking for `.xml.gz` still get the compressed bytes.
+- Doesn't match → `env` passes through to `@app.call`.
+
+## XSL stylesheets
+
+The middleware also serves the built-in XSL stylesheets — pretty sitemap rendering for human visitors — at their referenced paths. Configure their URLs via:
+
+```ruby
+configure do |config|
+  config.xsl_stylesheet_url       = '/_sitemap-stylesheet.xsl'
+  config.xsl_index_stylesheet_url = '/_sitemap-index-stylesheet.xsl'
+end
+```
+
+## Multi-tenant routing
+
+For per-tenant sitemaps stored under subpaths:
+
+```ruby
+use SiteMaps::Middleware,
+  adapter:        per_request_adapter,
+  storage_prefix: ->(request) { "tenants/#{request.host.split('.').first}" }
+```
+
+If the adapter itself already scopes paths by tenant, no prefix is needed — just point it at the right one for each request.
+
+## robots.txt integration
+
+Emit a `Sitemap:` directive for the generated file:
+
+```ruby
+# config.ru or a controller
+SiteMaps::RobotsTxt.sitemap_directive('https://example.com/sitemap.xml')
+# => "Sitemap: https://example.com/sitemap.xml"
+
+SiteMaps::RobotsTxt.render(
+  sitemap_url:      'https://example.com/sitemap.xml',
+  extra_directives: ['Disallow: /admin']
+)
+# => "Sitemap: https://example.com/sitemap.xml\nDisallow: /admin"
+```

data/docs/processes.md

@@ -0,0 +1,156 @@
+# Processes
+
+A **process** is a unit of work that produces part of a sitemap. Each process runs on its own thread, writes its own URL set, and becomes an entry in the sitemap index.
+
+## Static processes
+
+A static process has no parameters. It runs once and writes one (possibly split) sitemap file.
+
+```ruby
+SiteMaps.use(:file_system) do
+  configure { |c| c.url = 'https://example.com/sitemap.xml'; c.directory = 'public' }
+
+  process do |s|
+    s.add('/', priority: 1.0)
+    s.add('/about')
+  end
+
+  process :posts, 'posts/sitemap.xml' do |s|
+    Post.find_each { |post| s.add("/posts/#{post.slug}", lastmod: post.updated_at) }
+  end
+end
+```
+
+- Without an explicit name, the process is named `:default`.
+- Without an explicit location, a default filename is assigned.
+- The block receives a `SitemapBuilder` (`s`), on which `add` is called per URL.
+
+## Dynamic processes
+
+A dynamic process has placeholders in its location template and corresponding kwargs. Each unique combination of kwargs produces a separate sitemap file.
+
+```ruby
+process :monthly_posts, 'posts/%{year}-%{month}/sitemap.xml', year: 2024, month: 1 do |s, year:, month:, **|
+  Post.where('extract(year from published_at) = ? AND extract(month from published_at) = ?', year, month)
+      .find_each { |p| s.add("/posts/#{p.slug}", lastmod: p.updated_at) }
+end
+```
+
+The kwargs passed to `process` are **defaults**; the real values come from `Runner#enqueue`:
+
+```ruby
+runner = SiteMaps.generate(config_file: 'config/sitemap.rb')
+runner.enqueue(:monthly_posts, year: 2024, month: 1)
+runner.enqueue(:monthly_posts, year: 2024, month: 2)
+runner.enqueue(:monthly_posts, year: 2024, month: 3)
+runner.run
+```
+
+Or from the CLI:
+
+```bash
+bundle exec site_maps generate monthly_posts \
+  --config-file config/sitemap.rb \
+  --context=year:2024 month:1
+```
+
+## Execution model
+
+When you call `runner.run`:
+
+1. Each enqueued process is wrapped in a `Concurrent::Future`.
+2. The pool (default 4 threads, configurable via `--max-threads`) runs them in parallel.
+3. Each process builds a `URLSet`. When the set fills up (50,000 links, 1,000 news items, or 50 MB uncompressed), it's finalized and written, and a new URLSet starts — automatically.
+4. After every process finishes, the sitemap index is aggregated and written to `config.url`.
+
+## Splitting rules
+
+A URL set is finalized and rolled over when **any** of these apply:
+
+- Links reach `config.max_links` (default 50,000 — the sitemap spec limit).
+- News entries reach 1,000.
+- Uncompressed XML reaches 50 MB.
+
+Split files are named by `IncrementalLocation`: `posts/sitemap.xml` becomes `posts/sitemap1.xml`, `posts/sitemap2.xml`, etc.
+
+## Index generation
+
+A sitemap index is produced when:
+
+- More than one process exists,
+- A single process was split across multiple files, or
+- External sitemaps were added.
+
+Otherwise a single `urlset` is written directly at `config.url` (the "inline" optimization).
+
+## Adding external sitemaps
+
+Reference third-party or pre-existing sitemaps in the index:
+
+```ruby
+SiteMaps.use(:file_system) do
+  configure { |c| c.url = 'https://example.com/sitemap.xml'; c.directory = 'public' }
+
+  external_sitemap('https://cdn.example.com/legacy-sitemap.xml', lastmod: Time.parse('2024-01-15'))
+
+  process { |s| s.add('/') }
+end
+```
+
+## Shared helpers across processes
+
+Use `extend_processes_with` to add methods that every process block can call:
+
+```ruby
+module Helpers
+  def post_path(post) = "/posts/#{post.slug}"
+  def published_posts = Post.where.not(published_at: nil)
+end
+
+SiteMaps.use(:file_system) do
+  configure { |c| c.url = 'https://example.com/sitemap.xml'; c.directory = 'public' }
+  extend_processes_with(Helpers)
+
+  process :posts do |s|
+    published_posts.find_each { |p| s.add(post_path(p), lastmod: p.updated_at) }
+  end
+end
+```
+
+## URL filters
+
+Filters run per URL inside every process — use them for global exclusions or default attributes:
+
+```ruby
+SiteMaps.use(:file_system) do
+  configure { |c| c.url = 'https://example.com/sitemap.xml'; c.directory = 'public' }
+
+  # Exclude any /admin path
+  url_filter { |url, _options| false if url.include?('/admin') }
+
+  # Boost blog priority
+  url_filter do |url, options|
+    if url.include?('/blog/')
+      options.merge(priority: 0.9, changefreq: 'daily')
+    else
+      options
+    end
+  end
+
+  process { |s| ... }
+end
+```
+
+A filter returning `false` (or `nil`) excludes the URL entirely. Returning a hash replaces the options.
+
+## Re-running a single shard
+
+Only regenerate what changed — the rest is preserved from the existing sitemap index:
+
+```ruby
+runner = SiteMaps.generate(config_file: 'config/sitemap.rb')
+runner.enqueue(:monthly_posts, year: 2024, month: 3)  # only March
+runner.run                                             # Jan and Feb kept as-is
+```
+
+This is the main advantage of parameterized dynamic processes: you can rebuild one month's shard on a cron and leave the rest untouched.

data/docs/rails.md

@@ -0,0 +1,128 @@
+# Rails Integration
+
+The Railtie loads automatically when Rails is present. It wires two things:
+
+1. **URL helpers** — `s.route.<helper>` inside process blocks.
+2. **No other magic** — no initializer, no autoloaded directories, no patched generators.
+
+## URL helpers in processes
+
+```ruby
+# config/sitemap.rb
+SiteMaps.use(:file_system) do
+  configure do |config|
+    config.url       = 'https://example.com/sitemap.xml'
+    config.directory = Rails.public_path.to_s
+  end
+
+  process do |s|
+    s.add(s.route.root_path,  priority: 1.0)
+    s.add(s.route.about_path)
+    Post.find_each { |p| s.add(s.route.post_path(p), lastmod: p.updated_at) }
+  end
+end
+```
+
+`s.route` is a singleton wrapping `Rails.application.routes.url_helpers`.
+
+## Generating from Rails
+
+### One-off
+
+```bash
+bundle exec site_maps generate --config-file config/sitemap.rb
+```
+
+The CLI auto-requires `config/environment.rb` if it finds a `config/application.rb`, so ActiveRecord, URL helpers, and everything else loads as normal.
+
+### From a Rake task
+
+```ruby
+# lib/tasks/sitemap.rake
+namespace :sitemap do
+  desc 'Generate sitemaps'
+  task generate: :environment do
+    runner = SiteMaps.generate(config_file: Rails.root.join('config/sitemap.rb').to_s)
+    runner.enqueue_all.run
+  end
+end
+```
+
+Run on deploy or via cron:
+
+```bash
+bundle exec rake sitemap:generate
+```
+
+### From a scheduled job
+
+```ruby
+class SitemapJob < ApplicationJob
+  def perform
+    runner = SiteMaps.generate(config_file: Rails.root.join('config/sitemap.rb').to_s)
+    runner.enqueue_all.run
+  end
+end
+
+SitemapJob.set(cron: '0 3 * * *').perform_later
+```
+
+## Serving generated sitemaps
+
+Add the Rack middleware to serve files generated by the `:file_system` adapter:
+
+```ruby
+# config/application.rb
+config.middleware.use SiteMaps::Middleware, adapter: -> { SiteMaps.current_adapter }
+```
+
+See [middleware.md](middleware.md) for options.
+
+## Asset precompile integration
+
+If you want sitemaps regenerated on every deploy, hook into `assets:precompile`:
+
+```ruby
+# lib/tasks/sitemap.rake
+Rake::Task['assets:precompile'].enhance(['sitemap:generate'])
+```
+
+## robots.txt
+
+```erb
+<%# public/robots.txt.erb or app/views/robots.text.erb %>
+User-agent: *
+Disallow: /admin
+
+<%= SiteMaps::RobotsTxt.sitemap_directive('https://example.com/sitemap.xml') %>
+```
+
+## Multi-tenant
+
+`SiteMaps.define` gives you a generation function parameterized by runtime context:
+
+```ruby
+# config/sitemap.rb
+SiteMaps.define do |tenant:|
+  use(:file_system) do
+    configure do |config|
+      config.url       = "https://#{tenant.domain}/sitemap.xml"
+      config.directory = tenant.public_path
+    end
+
+    process { |s| tenant.pages.each { |page| s.add(page.path, lastmod: page.updated_at) } }
+  end
+end
+```
+
+```ruby
+Tenant.find_each do |tenant|
+  SiteMaps.generate(config_file: 'config/sitemap.rb', context: { tenant: tenant }).enqueue_all.run
+end
+```
+
+The context hash is splatted into the `define` block as keyword args.
+
+## Dependencies
+
+- Rails is **not** listed in the gemspec. The Railtie is loaded only if Rails is already present. If you're using `site_maps` in a non-Rails Ruby project, the Rails-specific pieces are inert.

data/lib/site_maps/adapters/adapter.rb

@@ -13,11 +13,12 @@ module SiteMaps::Adapters
     end
 
     def_delegators :config, :fetch_sitemap_index_links
-    attr_reader :sitemap_index, :processes
+    attr_reader :sitemap_index, :processes, :process_mixins
 
     def initialize(**options, &block)
       @config = SiteMaps.config.becomes(self.class.config_class, **options)
       @processes = Concurrent::Hash.new
+      @process_mixins = Concurrent::Array.new
       reset!
       instance_exec(&block) if block
     end
@@ -60,20 +61,49 @@ module SiteMaps::Adapters
       @processes[name] = SiteMaps::Process.new(name, location, kwargs, block)
     end
 
+    def external_sitemap(url, lastmod: nil)
+      @external_sitemaps ||= Concurrent::Array.new
+      @external_sitemaps << SiteMaps::Builder::SitemapIndex::Item.new(url, lastmod)
+    end
+
+    def external_sitemaps
+      @external_sitemaps || []
+    end
+
     def maybe_inline_urlset?
-      @processes.size == 1 && @processes.first.last.static?
+      @processes.size == 1 && @processes.first.last.static? && external_sitemaps.empty?
     end
 
     def repo
       @repo ||= SiteMaps::AtomicRepository.new(config.url)
     end
 
-    def include_module(mod)
-      extend(mod)
+    def extend_processes_with(mod)
+      @process_mixins << mod
+    end
+
+    def url_filter(&block)
+      if block
+        @url_filters ||= Concurrent::Array.new
+        @url_filters << block
+      end
+      @url_filters || []
+    end
+
+    def apply_url_filters(link, options)
+      url = link.respond_to?(:to_s) ? link.to_s : link
+      url_filter.each do |filter|
+        result = filter.call(url, options)
+        return nil if result == false
+
+        options = result if result.is_a?(Hash)
+      end
+      options
     end
 
     def reset!
-      @sitemap_index = SiteMaps::Builder::SitemapIndex.new
+      xsl_url = config.respond_to?(:xsl_index_stylesheet_url) ? config.xsl_index_stylesheet_url : nil
+      @sitemap_index = SiteMaps::Builder::SitemapIndex.new(xsl_url: xsl_url)
       @repo = nil
     end
   end

data/lib/site_maps/adapters/aws_sdk/storage.rb

@@ -14,8 +14,7 @@ class SiteMaps::Adapters::AwsSdk::Storage
     lastmod = options.delete(:last_modified) || Time.now
     options[:metadata] ||= {}
     options[:metadata]["given-last-modified"] = lastmod.utc.strftime("%Y-%m-%dT%H:%M:%S%:z")
-    obj = object(location.remote_path)
-    obj.upload_file(location.path, **options)
+    transfer_manager.upload_file(location.path, bucket: config.bucket, key: location.remote_path, **options)
   end
 
   def read(location)
@@ -49,4 +48,8 @@ class SiteMaps::Adapters::AwsSdk::Storage
   def object(remote_path)
     config.s3_bucket.object(remote_path)
   end
+
+  def transfer_manager
+    @transfer_manager ||= ::Aws::S3::TransferManager.new(client: config.s3_resource.client)
+  end
 end

data/lib/site_maps/builder/sitemap_index/item.rb

@@ -25,7 +25,7 @@ class SiteMaps::Builder::SitemapIndex::Item < Struct.new(:loc, :lastmod)
     return unless loc =~ %r{^https?://[^/]+(/.*)$}
 
     val = File.dirname(Regexp.last_match(1))
-    val = val[1..-1] if val.start_with?("/")
+    val = val[1..] if val.start_with?("/")
     val
   end
 

data/lib/site_maps/builder/sitemap_index.rb

@@ -2,20 +2,22 @@
 
 module SiteMaps::Builder
   class SitemapIndex
-    HEADER = <<~HEADER
-      <?xml version="1.0" encoding="UTF-8"?>
+    XML_DECLARATION = %(<?xml version="1.0" encoding="UTF-8"?>)
+    SITEMAPINDEX_OPEN = <<~SITEMAPINDEX_OPEN
       <sitemapindex
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/siteindex.xsd"
         xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
       >
-    HEADER
+    SITEMAPINDEX_OPEN
+    HEADER = "#{XML_DECLARATION}\n#{SITEMAPINDEX_OPEN}"
     FOOTER = "</sitemapindex>"
 
     attr_reader :sitemaps
 
-    def initialize
+    def initialize(xsl_url: nil)
       @sitemaps = Concurrent::Set.new
+      @xsl_url = xsl_url
     end
 
     def add(loc, lastmod: nil)
@@ -25,7 +27,13 @@ module SiteMaps::Builder
 
     def to_xml
       io = StringIO.new
-      io.puts(HEADER)
+      if @xsl_url
+        io.puts(XML_DECLARATION)
+        io.puts(XSLStylesheet.processing_instruction(@xsl_url))
+        io.puts(SITEMAPINDEX_OPEN)
+      else
+        io.puts(HEADER)
+      end
       @sitemaps.each do |sitemap|
         io.puts(sitemap.to_xml)
       end
@@ -33,8 +41,24 @@ module SiteMaps::Builder
       io.string
     end
 
+    def last_modified
+      dates = @sitemaps.filter_map { |s| parse_lastmod(s.lastmod) }
+      dates.max || Time.now
+    end
+
     def empty?
       @sitemaps.empty?
     end
+
+    private
+
+    def parse_lastmod(value)
+      case value
+      when Time then value
+      when String then Time.parse(value)
+      end
+    rescue ArgumentError
+      nil
+    end
   end
 end