airblade-sitemap_generator 0.3.4

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 [name of plugin creator]
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,232 @@
+N.B. This fork is Ruby 1.8.6-compatible (and probably therefore not compatible with 1.8.7+).
+
+<hr>
+
+SitemapGenerator
+================
+
+A Rails 3-compatible gem/plugin to generate ['enterprise-class'][enterprise_class] Sitemaps using a familiar Rails Routes-like DSL.  Sitemaps are readable by all search engines and adhere to the ['Sitemap protocol specification'][sitemap_protocol].  Automatically pings search engines to notify them of new sitemaps (including Google, Yahoo and Bing).  Provides rake tasks to easily manage your sitemaps.  Supports image sitemaps and handles millions of links.
+
+Features
+-------
+
+- v0.2.6: **Support ['image sitemaps'][sitemap_images]**!
+- v0.2.5: **Support Rails 3**!
+
+- Adheres to the ['Sitemap protocol specification'][sitemap_protocol]
+- Handles millions of links
+- Automatic Gzip of Sitemap files
+- Automatic ping of search engines to notify them of new sitemaps: Google, Yahoo, Bing, Ask, SitemapWriter
+- Won't clobber your old sitemaps if the new one fails to generate
+- Set the priority of links, change frequency etc
+- You control which links are included
+- You set the host name, so it doesn't matter if your application is in a subdirectory
+
+Foreword
+-------
+
+Unfortunately, Adam Salter passed away in 2009.  Those who knew him know what an amazing guy he was, and what an excellent Rails programmer he was.  His passing is a great loss to the Rails community.
+
+[Karl Varga](http://github.com/kjvarga) has taken over development of SitemapGenerator.  The canonical repository is [http://github.com/kjvarga/sitemap_generator][canonical_repo]
+
+Installation
+=======
+
+**Rails 3:**
+
+1. Add the gem to your <tt>Gemspec</tt>
+
+    <code>gem 'sitemap_generator'</code>
+
+2. `$ rake sitemap:install`
+
+**Rails 2.x: As a gem**
+
+1. Add the gem as a dependency in your <tt>config/environment.rb</tt>
+
+    <code>config.gem 'sitemap_generator', :lib => false</code>
+
+2. `$ rake gems:install`
+
+3. Add the following to your <tt>RAILS_ROOT/Rakefile</tt>
+
+    <pre>begin
+      require 'sitemap_generator/tasks'
+    rescue Exception => e
+      puts "Warning, couldn't load gem tasks: #{e.message}! Skipping..."
+    end</pre>
+
+4. `$ rake sitemap:install`
+
+**Rails 2.x: As a plugin**
+
+1. <code>$ ./script/plugin install git://github.com/kjvarga/sitemap_generator.git</code>
+
+----
+
+Installation creates a <tt>config/sitemap.rb</tt> file which will contain your logic for generating the Sitemap files.  If you want to create this file manually run <code>rake sitemap:install</code>.
+
+You can run <code>rake sitemap:refresh</code> as needed to create Sitemap files. This will also ping these ['major search engines'][sitemap_engines]: Google, Yahoo, Bing, Ask, SitemapWriter.  If you want to disable all non-essential output run the rake task with <code>rake -s sitemap:refresh</code>.
+
+To keep your Sitemaps up-to-date, setup a cron job.  Pass the <tt>-s</tt> option to the rake task to silence all but the most important output.  If you're using Whenever, then your schedule would look something like:
+
+    # config/schedule.rb
+    every 1.day, :at => '5:00 am' do
+      rake "-s sitemap:refresh"
+    end
+
+Optionally, you can add the following to your <code>public/robots.txt</code> file, so that robots can find the sitemap file:
+
+    Sitemap: <hostname>/sitemap_index.xml.gz
+
+The Sitemap URL in the robots file should be the complete URL to the Sitemap Index, such as <tt>http://www.example.org/sitemap_index.xml.gz</tt>
+
+
+Example 'config/sitemap.rb'
+==========
+
+    # Set the host name for URL creation
+    SitemapGenerator::Sitemap.default_host = "http://www.example.com"
+
+    SitemapGenerator::Sitemap.add_links do |sitemap|
+      # Put links creation logic here.
+      #
+      # The Root Path ('/') and Sitemap Index file are added automatically.
+      # Links are added to the Sitemap output in the order they are specified.
+      #
+      # Usage: sitemap.add path, options
+      #        (default options are used if you don't specify them)
+      #
+      # Defaults: :priority => 0.5, :changefreq => 'weekly',
+      #           :lastmod => Time.now, :host => default_host
+
+
+      # Examples:
+
+      # add '/articles'
+      sitemap.add articles_path, :priority => 0.7, :changefreq => 'daily'
+
+      # add all individual articles
+      Article.find(:all).each do |a|
+        sitemap.add article_path(a), :lastmod => a.updated_at
+      end
+
+      # add merchant path
+      sitemap.add '/purchase', :priority => 0.7, :host => "https://www.example.com"
+
+      # add all individual news with images
+      News.all.each do |n|
+        sitemap.add news_path(n), :lastmod => n.updated_at, :images=>n.images.collect{ |r| :loc=>r.image.url, :title=>r.image.name }
+      end
+
+    end
+
+    # Including Sitemaps from Rails Engines.
+    #
+    # These Sitemaps should be almost identical to a regular Sitemap file except
+    # they needn't define their own SitemapGenerator::Sitemap.default_host since
+    # they will undoubtedly share the host name of the application they belong to.
+    #
+    # As an example, say we have a Rails Engine in vendor/plugins/cadability_client
+    # We can include its Sitemap here as follows:
+    #
+    file = File.join(Rails.root, 'vendor/plugins/cadability_client/config/sitemap.rb')
+    eval(open(file).read, binding, file)
+
+Raison d'être
+-------
+
+Most of the Sitemap plugins out there seem to try to recreate the Sitemap links by iterating the Rails routes. In some cases this is possible, but for a great deal of cases it isn't.
+
+a) There are probably quite a few routes in your routes file that don't need inclusion in the Sitemap. (AJAX routes I'm looking at you.)
+
+and
+
+b) How would you infer the correct series of links for the following route?
+
+    map.zipcode 'location/:state/:city/:zipcode', :controller => 'zipcode', :action => 'index'
+
+Don't tell me it's trivial, because it isn't. It just looks trivial.
+
+So my idea is to have another file similar to 'routes.rb' called 'sitemap.rb', where you can define what goes into the Sitemap.
+
+Here's my solution:
+
+    Zipcode.find(:all, :include => :city).each do |z|
+      sitemap.add zipcode_path(:state => z.city.state, :city => z.city, :zipcode => z)
+    end
+
+Easy hey?
+
+Other Sitemap settings for the link, like `lastmod`, `priority`, `changefreq` and `host` are entered automatically, although you can override them if you need to.
+
+Compatibility
+=======
+
+Tested and working on:
+
+- **Rails** 3.0.0, sitemap_generator version >= 0.2.5
+- **Rails** 1.x - 2.3.5
+- **Ruby** 1.8.7, 1.9.1
+
+Notes
+=======
+
+1) For large sitemaps it may be useful to split your generation into batches to avoid running out of memory. E.g.:
+
+    # add movies
+    Movie.find_in_batches(:batch_size => 1000) do |movies|
+      movies.each do |movie|
+        sitemap.add "/movies/show/#{movie.to_param}", :lastmod => movie.updated_at, :changefreq => 'weekly'
+      end
+    end
+
+2) New Capistrano deploys will remove your Sitemap files, unless you run `rake sitemap:refresh`. The way around this is to create a cap task:
+
+    after "deploy:update_code", "deploy:copy_old_sitemap"
+
+    namespace :deploy do
+      task :copy_old_sitemap do
+          run "if [ -e #{previous_release}/public/sitemap_index.xml.gz ]; then cp #{previous_release}/public/sitemap* #{current_release}/public/; fi"
+      end
+    end
+
+3) If generation of your sitemap fails for some reason, the old sitemap will remain in public/.  This ensures that robots will always find a valid sitemap.  Running silently (`rake -s sitemap:refresh`) and with email forwarding setup you'll only get an email if your sitemap fails to build, and no notification when everything is fine - which will be most of the time.
+
+Known Bugs
+========
+
+- There's no check on the size of a URL which [isn't supposed to exceed 2,048 bytes][sitemaps_xml].
+- Currently only supports one Sitemap Index file, which can contain 50,000 Sitemap files which can each contain 50,000 urls, so it _only_ supports up to 2,500,000,000 (2.5 billion) urls. I personally have no need of support for more urls, but plugin could be improved to support this.
+
+Wishlist & Coming Soon
+========
+
+- Support for generating sitemaps for sites with multiple domains.  Sitemaps are generated into subdirectories and we use a Rack middleware to rewrite requests for sitemaps to the correct subdirectory based on the request host.
+- I want to refactor the code because it has grown a lot.  Part of this refactoring will include implementing some more checks to make sure we adhere to standards as well as making sure that the sitemaps are being generated as efficiently as possible.
+
+I'd like to simplify adding links to a sitemap.  Right now it's all or nothing.  I'd like to break it up so you can add batches.
+- Auto coverage testing.  Generate a report of broken URLs by checking the status codes of each page in the sitemap.
+
+Thanks (in no particular order)
+========
+
+- [Alexadre Bini](http://github.com/alexandrebini) for image sitemaps
+- [Dan Pickett](http://github.com/dpickett)
+- [Rob Biedenharn](http://github.com/rab)
+- [Richie Vos](http://github.com/jerryvos)
+- [Adrian Mugnolo](http://github.com/xymbol)
+- [Jason Weathered](http://github.com/jasoncodes)
+
+Copyright (c) 2009 Karl Varga released under the MIT license
+
+[canonical_repo]:http://github.com/kjvarga/sitemap_generator
+[enterprise_class]:https://twitter.com/dhh/status/1631034662 "I use enterprise in the same sense the Phusion guys do - i.e. Enterprise Ruby. Please don't look down on my use of the word 'enterprise' to represent being a cut above. It doesn't mean you ever have to work for a company the size of IBM. Or constantly fight inertia, writing crappy software, adhering to change management practices and spending hours in meetings... Not that there's anything wrong with that - Wait, what?"
+[sitemap_engines]:http://en.wikipedia.org/wiki/Sitemap_index "http://en.wikipedia.org/wiki/Sitemap_index"
+[sitemaps_org]:http://www.sitemaps.org/protocol.php "http://www.sitemaps.org/protocol.php"
+[sitemaps_xml]:http://www.sitemaps.org/protocol.php#xmlTagDefinitions "XML Tag Definitions"
+[sitemap_generator_usage]:http://wiki.github.com/adamsalter/sitemap_generator/sitemapgenerator-usage "http://wiki.github.com/adamsalter/sitemap_generator/sitemapgenerator-usage"
+[boost_juice]:http://www.boostjuice.com.au/ "Mmmm, sweet, sweet Boost Juice."
+[cb]:http://codebright.net "http://codebright.net"
+[sitemap_images]:http://www.google.com/support/webmasters/bin/answer.py?answer=178636
+[sitemap_protocol]:http://sitemaps.org/protocol.php

data/Rakefile

@@ -0,0 +1,114 @@
+require 'rake'
+require 'rake/rdoctask'
+require 'rubygems'
+gem 'rspec', '1.3.0'
+require 'spec/rake/spectask'
+gem 'nokogiri'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    gem.name = "airblade-sitemap_generator"
+    gem.summary = %Q{Easily generate enterprise class Sitemaps for your Rails site using a familiar Rails Routes-like DSL}
+    gem.description = %Q{A Rails 3-compatible gem/plugin to generate enterprise-class Sitemaps using a familiar Rails Routes-like DSL.  Sitemaps are readable by all search engines and adhere to the Sitemap protocol specification.  Automatically pings search engines to notify them of new sitemaps (including Google, Yahoo and Bing).  Provides rake tasks to easily manage your sitemaps.  Supports image sitemaps and handles millions of links.}
+    gem.email = "boss@airbladesoftware.com"
+    gem.homepage = "http://github.com/airblade/sitemap_generator"
+    gem.authors = ["Adam Salter", "Karl Varga"]
+    gem.files =  FileList["[A-Z]*", "{bin,lib,rails,templates,tasks}/**/*"]
+    gem.test_files = []
+    gem.add_development_dependency "rspec"
+    gem.add_development_dependency "nokogiri"
+  end
+  Jeweler::GemcutterTasks.new
+rescue LoadError
+  puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
+end
+
+#
+# Helper methods
+#
+module Helpers
+  extend self
+
+  # Return a full local path to path fragment <tt>path</tt>
+  def local_path(path)
+    File.join(File.dirname(__FILE__), path)
+  end
+
+  # Copy all of the local files into <tt>path</tt> after completely cleaning it
+  def prepare_path(path)
+    rm_rf path
+    mkdir_p path
+    cp_r(FileList["[A-Z]*", "{bin,lib,rails,templates,tasks}"], path)
+  end
+end
+
+#
+# Tasks
+#
+task :default => :test
+
+namespace :test do
+  #desc "Test as a gem, plugin and Rails 3 gem"
+  #task :all => ['test:gem', 'test:plugin']
+
+  task :gem => ['test:prepare:gem', 'multi_spec']
+  task :plugin => ['test:prepare:plugin', 'multi_spec']
+  task :rails3 => ['test:prepare:rails3', 'multi_spec']
+
+  task :multi_spec do
+    Rake::Task['spec'].invoke
+    Rake::Task['spec'].reenable
+  end
+
+  namespace :prepare do
+    task :gem do
+      ENV["SITEMAP_RAILS"] = 'gem'
+      Helpers.prepare_path(Helpers.local_path('spec/mock_app_gem/vendor/gems/sitemap_generator-1.2.3'))
+      rm_rf(Helpers.local_path('spec/mock_app_gem/public/sitemap*'))
+    end
+
+    task :plugin do
+      ENV["SITEMAP_RAILS"] = 'plugin'
+      Helpers.prepare_path(Helpers.local_path('spec/mock_app_plugin/vendor/plugins/sitemap_generator-1.2.3'))
+      rm_rf(Helpers.local_path('spec/mock_app_plugin/public/sitemap*'))
+    end
+
+    task :rails3 do
+      ENV["SITEMAP_RAILS"] = 'rails3'
+      rm_rf(Helpers.local_path('spec/mock_rails3_gem/public/sitemap*'))
+    end
+  end
+end
+
+desc "Release a new patch version"
+task :release_new_version do
+  Rake::Task['version:bump:patch'].invoke
+  Rake::Task['github:release'].invoke
+  Rake::Task['git:release'].invoke
+  Rake::Task['gemcutter:release'].invoke
+end
+
+desc "Run tests as a gem install"
+task :test => ['test:gem']
+
+Spec::Rake::SpecTask.new(:spec) do |spec|
+  spec.libs << 'lib' << 'spec'
+  spec.spec_files = FileList['spec/**/*_spec.rb']
+end
+task :spec => :check_dependencies
+
+Spec::Rake::SpecTask.new(:rcov) do |spec|
+  spec.libs << 'lib' << 'spec'
+  spec.pattern = 'spec/**/*_spec.rb'
+  spec.rcov = true
+end
+
+desc 'Generate documentation'
+Rake::RDocTask.new(:rdoc) do |rdoc|
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title    = 'SitemapGenerator'
+  rdoc.options << '--line-numbers' << '--inline-source'
+  rdoc.rdoc_files.include('README.md')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end

data/VERSION

@@ -0,0 +1 @@
+0.3.4

data/lib/sitemap_generator.rb

@@ -0,0 +1,28 @@
+require 'sitemap_generator/builder'
+require 'sitemap_generator/mapper'
+require 'sitemap_generator/link'
+require 'sitemap_generator/link_set'
+require 'sitemap_generator/templates'
+require 'sitemap_generator/utilities'
+require 'sitemap_generator/railtie' if SitemapGenerator::Utilities.rails3?
+
+require 'active_support/core_ext/numeric'
+
+module SitemapGenerator
+  silence_warnings do
+    VERSION = File.read(File.dirname(__FILE__) + "/../VERSION").strip
+    MAX_SITEMAP_FILES    = 50_000        # max sitemap links per index file
+    MAX_SITEMAP_LINKS    = 50_000        # max links per sitemap
+    MAX_SITEMAP_IMAGES   = 1_000         # max images per url
+    MAX_SITEMAP_FILESIZE = 10.megabytes  # bytes
+
+    Sitemap = LinkSet.new
+  end
+
+  class << self
+    attr_accessor :root, :templates
+  end
+
+  self.root = File.expand_path(File.join(File.dirname(__FILE__), '../'))
+  self.templates = SitemapGenerator::Templates.new(self.root)
+end

data/lib/sitemap_generator/builder.rb

@@ -0,0 +1,9 @@
+require 'sitemap_generator/builder/helper'
+require 'sitemap_generator/builder/sitemap_file'
+require 'sitemap_generator/builder/sitemap_index_file'
+
+module SitemapGenerator
+  module Builder
+
+  end
+end

data/lib/sitemap_generator/builder/helper.rb

@@ -0,0 +1,10 @@
+module SitemapGenerator
+  module Builder
+    module Helper
+
+      def w3c_date(date)
+         date.utc.strftime("%Y-%m-%dT%H:%M:%S+00:00")
+      end
+    end
+  end
+end

data/lib/sitemap_generator/builder/sitemap_file.rb

@@ -0,0 +1,124 @@
+require 'sitemap_generator/builder/helper'
+require 'builder'
+require 'zlib'
+
+module SitemapGenerator
+  module Builder
+    class SitemapFile
+      include SitemapGenerator::Builder::Helper
+
+      attr_accessor :sitemap_path, :public_path, :filesize, :link_count, :hostname
+
+      # <tt>public_path</tt> full path of the directory to write sitemaps in.
+      #   Usually your Rails <tt>public/</tt> directory.
+      #
+      # <tt>sitemap_path</tt> relative path including filename of the sitemap
+      #   file relative to <tt>public_path</tt>
+      #
+      # <tt>hostname</tt> hostname including protocol to use in all links
+      #   e.g. http://en.google.ca
+      def initialize(public_path, sitemap_path, hostname)
+        self.sitemap_path = sitemap_path
+        self.public_path = public_path
+        self.hostname = hostname
+        self.link_count = 0
+
+        @xml_content       = ''     # XML urlset content
+        @xml_wrapper_start = <<-HTML
+          <?xml version="1.0" encoding="UTF-8"?>
+            <urlset
+              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+              xmlns:image="http://www.google.com/schemas/sitemap-image/1.1"
+              xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9
+                http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd"
+              xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
+            >
+        HTML
+        @xml_wrapper_start.gsub!(/\s+/, ' ').gsub!(/ *> */, '>').strip!
+        @xml_wrapper_end   = %q[</urlset>]
+        self.filesize = @xml_wrapper_start.length + @xml_wrapper_end.length
+      end
+
+      def lastmod
+        File.mtime(self.full_path) rescue nil
+      end
+
+      def empty?
+        self.link_count == 0
+      end
+
+      def full_url
+        URI.join(self.hostname, self.sitemap_path).to_s
+      end
+
+      def full_path
+        @full_path ||= File.join(self.public_path, self.sitemap_path)
+      end
+
+      # Return a boolean indicating whether the sitemap file can fit another link
+      # of <tt>bytes</tt> bytes in size.
+      def file_can_fit?(bytes)
+        (self.filesize + bytes) < SitemapGenerator::MAX_SITEMAP_FILESIZE && self.link_count < SitemapGenerator::MAX_SITEMAP_LINKS
+      end
+
+      # Add a link to the sitemap file and return a boolean indicating whether the
+      # link was added.
+      #
+      # If a link cannot be added, the file is too large or the link limit has been reached.
+      def add_link(link)
+        xml = build_xml(::Builder::XmlMarkup.new, link)
+        unless file_can_fit?(xml.length)
+          self.finalize!
+          return false
+        end
+
+        @xml_content << xml
+        self.filesize += xml.length
+        self.link_count += 1
+        true
+      end
+      alias_method :<<, :add_link
+
+      # Return XML as a String
+      def build_xml(builder, link)
+        builder.url do
+          builder.loc        link[:loc]
+          builder.lastmod    w3c_date(link[:lastmod])   if link[:lastmod]
+          builder.changefreq link[:changefreq]          if link[:changefreq]
+          builder.priority   link[:priority]            if link[:priority]
+
+          unless link[:images].blank?
+            link[:images].each do |image|
+              builder.image:image do
+                builder.image :loc, image[:loc]
+                builder.image :caption, image[:caption]             if image[:caption]
+                builder.image :geo_location, image[:geo_location]   if image[:geo_location]
+                builder.image :title, image[:title]                 if image[:title]
+                builder.image :license, image[:license]             if image[:license]
+              end
+            end
+          end
+        end
+        builder << ''
+      end
+
+      # Insert the content into the XML "wrapper" and write and close the file.
+      #
+      # All the xml content in the instance is cleared, but attributes like
+      # <tt>filesize</tt> are still available.
+      def finalize!
+        return if self.frozen?
+
+        open(self.full_path, 'wb') do |file|
+          gz = Zlib::GzipWriter.new(file)
+          gz.write @xml_wrapper_start
+          gz.write @xml_content
+          gz.write @xml_wrapper_end
+          gz.close
+        end
+        @xml_content = @xml_wrapper_start = @xml_wrapper_end = ''
+        self.freeze
+      end
+    end
+  end
+end

data/lib/sitemap_generator/builder/sitemap_index_file.rb

@@ -0,0 +1,33 @@
+module SitemapGenerator
+  module Builder
+    class SitemapIndexFile < SitemapFile
+
+      def initialize(*args)
+        super(*args)
+
+        @xml_content = '' # XML urlset content
+        @xml_wrapper_start = <<-HTML
+          <?xml version="1.0" encoding="UTF-8"?>
+            <sitemapindex
+              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+              xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9
+                http://www.sitemaps.org/schemas/sitemap/0.9/siteindex.xsd"
+              xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
+            >
+        HTML
+        @xml_wrapper_start.gsub!(/\s+/, ' ').gsub!(/ *> */, '>').strip!
+        @xml_wrapper_end   = %q[</sitemapindex>]
+        self.filesize = @xml_wrapper_start.length + @xml_wrapper_end.length
+      end
+
+      # Return XML as a String
+      def build_xml(builder, link)
+        builder.sitemap do
+          builder.loc        link[:loc]
+          builder.lastmod    w3c_date(link[:lastmod])   if link[:lastmod]
+        end
+        builder << ''
+      end
+    end
+  end
+end

data/lib/sitemap_generator/interpreter.rb

@@ -0,0 +1,28 @@
+module SitemapGenerator
+
+  # Evaluate a sitemap config file within the context of a class that includes the
+  # Rails URL helpers.
+  class Interpreter
+
+    if SitemapGenerator::Utilities.rails3?
+      include ::Rails.application.routes.url_helpers
+    else
+      require 'action_controller'
+      include ActionController::UrlWriter
+    end
+
+    def initialize(sitemap_config_file=nil)
+      sitemap_config_file ||= File.join(::Rails.root, 'config/sitemap.rb')
+      eval(open(sitemap_config_file).read)
+    end
+
+    # KJV do we need this?  We should be using path_* helpers.
+    # def self.default_url_options(options = nil)
+    #   { :host => SitemapGenerator::Sitemap.default_host }
+    # end
+
+    def self.run
+      new
+    end
+  end
+end

data/lib/sitemap_generator/link.rb

@@ -0,0 +1,36 @@
+module SitemapGenerator
+  module Link
+    extend self
+
+    # Return a Hash of options suitable to pass to a SitemapGenerator::Builder::SitemapFile instance.
+    def generate(path, options = {})
+      if path.is_a?(SitemapGenerator::Builder::SitemapFile)
+        options.reverse_merge!(:host => path.hostname, :lastmod => path.lastmod)
+        path = path.sitemap_path
+      end
+
+      options.assert_valid_keys(:priority, :changefreq, :lastmod, :host, :images)
+      options.reverse_merge!(:priority => 0.5, :changefreq => 'weekly', :lastmod => Time.now, :host => Sitemap.default_host, :images => [])
+      {
+        :path => path,
+        :priority => options[:priority],
+        :changefreq => options[:changefreq],
+        :lastmod => options[:lastmod],
+        :host => options[:host],
+        :loc => URI.join(options[:host], path).to_s,
+        :images => prepare_images(options[:images], options[:host])
+      }
+    end
+
+    # Return an Array of image option Hashes suitable to be parsed by SitemapGenerator::Builder::SitemapFile
+    def prepare_images(images, host)
+      images.delete_if { |key,value| key[:loc] == nil }
+      images.each do |r|
+        r.assert_valid_keys(:loc, :caption, :geo_location, :title, :license)
+        r[:loc] = URI.join(host, r[:loc]).to_s
+      end
+      images[0..(SitemapGenerator::MAX_SITEMAP_IMAGES-1)]
+    end
+  end
+end
+

data/lib/sitemap_generator/link_set.rb

@@ -0,0 +1,174 @@
+require 'builder'
+require 'action_view'
+
+# A LinkSet provisions a bunch of links to sitemap files.  It also writes the index file
+# which lists all the sitemap files written.
+module SitemapGenerator
+  class LinkSet
+    include ActionView::Helpers::NumberHelper  # for number_with_delimiter
+
+    attr_accessor :default_host, :public_path, :sitemaps_path
+    attr_accessor :sitemap, :sitemaps, :sitemap_index
+    attr_accessor :verbose, :yahoo_app_id
+
+    # Evaluate the sitemap config file and write all sitemaps.
+    #
+    # This should be refactored so that we can have multiple instances
+    # of LinkSet.
+    def create
+      require 'sitemap_generator/interpreter'
+
+      self.public_path = File.join(::Rails.root, 'public/') if self.public_path.nil?
+
+      start_time = Time.now
+      SitemapGenerator::Interpreter.run
+      finalize!
+      end_time = Time.now
+
+      puts "\nSitemap stats: #{number_with_delimiter(self.link_count)} links / #{self.sitemaps.size} files / " + ("%dm%02ds" % (end_time - start_time).divmod(60)) if verbose
+    end
+
+    # <tt>public_path</tt> (optional) full path to the directory to write sitemaps in.
+    #   Defaults to your Rails <tt>public/</tt> directory.
+    #
+    # <tt>sitemaps_path</tt> (optional) path fragment within public to write sitemaps
+    #   to e.g. 'en/'.  Sitemaps are written to <tt>public_path</tt> + <tt>sitemaps_path</tt>
+    #
+    # <tt>default_host</tt> hostname including protocol to use in all sitemap links
+    #   e.g. http://en.google.ca
+    def initialize(public_path = nil, sitemaps_path = nil, default_host = nil)
+      self.default_host = default_host
+      self.public_path = public_path
+      self.sitemaps_path = sitemaps_path
+
+      # Completed sitemaps
+      self.sitemaps = []
+    end
+
+    def link_count
+      self.sitemaps.inject(0) { |link_count_sum, sitemap| link_count_sum + sitemap.link_count }
+    end
+
+    # Called within the user's eval'ed sitemap config file.  Add links to sitemap files
+    # passing a block.
+    #
+    # TODO: Refactor.  The call chain is confusing and convoluted here.
+    def add_links
+      raise ArgumentError, "Default hostname not set" if default_host.blank?
+
+      # I'd rather have these calls in <tt>create</tt> but we have to wait
+      # for <tt>default_host</tt> to be set by the user's sitemap config
+      new_sitemap
+      add_default_links
+
+      yield Mapper.new(self)
+    end
+
+    # Called from Mapper.
+    #
+    # Add a link to the current sitemap.
+    def add_link(link)
+      unless self.sitemap << link
+        new_sitemap
+        self.sitemap << link
+      end
+    end
+
+    # Add the current sitemap to the <tt>sitemaps</tt> Array and
+    # start a new sitemap.
+    #
+    # If the current sitemap is nil or empty it is not added.
+    def new_sitemap
+      unless self.sitemap_index
+        self.sitemap_index = SitemapGenerator::Builder::SitemapIndexFile.new(public_path, sitemap_index_path, default_host)
+      end
+
+      unless self.sitemap
+        self.sitemap = SitemapGenerator::Builder::SitemapFile.new(public_path, new_sitemap_path, default_host)
+      end
+
+      # Mark the sitemap as complete and add it to the sitemap index
+      unless self.sitemap.empty?
+        self.sitemap.finalize!
+        self.sitemap_index << Link.generate(self.sitemap)
+        self.sitemaps << self.sitemap
+        show_progress(self.sitemap) if verbose
+
+        self.sitemap = SitemapGenerator::Builder::SitemapFile.new(public_path, new_sitemap_path, default_host)
+      end
+    end
+
+    # Report progress line.
+    def show_progress(sitemap)
+      uncompressed_size = number_to_human_size(sitemap.filesize)
+      compressed_size =   number_to_human_size(File.size?(sitemap.full_path))
+      puts "+ #{sitemap.sitemap_path}   #{sitemap.link_count} links / #{uncompressed_size} / #{compressed_size} gzipped"
+    end
+
+    # Finalize all sitemap files
+    def finalize!
+      new_sitemap
+      self.sitemap_index.finalize!
+    end
+
+    # Ping search engines.
+    #
+    # @see http://en.wikipedia.org/wiki/Sitemap_index
+    def ping_search_engines
+      require 'open-uri'
+
+      sitemap_index_url = CGI.escape(self.sitemap_index.full_url)
+      search_engines = {
+        :google         => "http://www.google.com/webmasters/sitemaps/ping?sitemap=#{sitemap_index_url}",
+        :yahoo          => "http://search.yahooapis.com/SiteExplorerService/V1/ping?sitemap=#{sitemap_index_url}&appid=#{yahoo_app_id}",
+        :ask            => "http://submissions.ask.com/ping?sitemap=#{sitemap_index_url}",
+        :bing           => "http://www.bing.com/webmaster/ping.aspx?siteMap=#{sitemap_index_url}",
+        :sitemap_writer => "http://www.sitemapwriter.com/notify.php?crawler=all&url=#{sitemap_index_url}"
+      }
+
+      puts "\n" if verbose
+      search_engines.each do |engine, link|
+        next if engine == :yahoo && !self.yahoo_app_id
+        begin
+          open(link)
+          puts "Successful ping of #{engine.to_s.titleize}" if verbose
+        rescue Timeout::Error, StandardError => e
+          puts "Ping failed for #{engine.to_s.titleize}: #{e.inspect} (URL #{link})" if verbose
+        end
+      end
+
+      if !self.yahoo_app_id && verbose
+        puts "\n"
+        puts <<-END.gsub(/^\s+/, '')
+          To ping Yahoo you require a Yahoo AppID.  Add it to your config/sitemap.rb with:
+
+          SitemapGenerator::Sitemap.yahoo_app_id = "my_app_id"
+
+          For more information see http://developer.yahoo.com/search/siteexplorer/V1/updateNotification.html
+        END
+      end
+    end
+
+    protected
+
+    def add_default_links
+      self.sitemap << Link.generate('/', :lastmod => Time.now, :changefreq => 'always', :priority => 1.0)
+      self.sitemap << Link.generate(self.sitemap_index, :lastmod => Time.now, :changefreq => 'always', :priority => 1.0)
+    end
+
+    # Return the current sitemap filename with index.
+    #
+    # The index depends on the length of the <tt>sitemaps</tt> array.
+    def new_sitemap_path
+      File.join(self.sitemaps_path || '', "sitemap#{self.sitemaps.length + 1}.xml.gz")
+    end
+
+    # Return the current sitemap index filename.
+    #
+    # At the moment we only support one index file which can link to
+    # up to 50,000 sitemap files.
+    def sitemap_index_path
+      File.join(self.sitemaps_path || '', 'sitemap_index.xml.gz')
+    end
+  end
+end

data/lib/sitemap_generator/mapper.rb

@@ -0,0 +1,16 @@
+module SitemapGenerator
+  # Generator instances are used to build links.
+  # The object passed to the add_links block in config/sitemap.rb is a Generator instance.
+  class Mapper
+    attr_accessor :set
+
+    def initialize(set)
+      @set = set
+    end
+
+    def add(loc, options = {})
+      set.add_link Link.generate(loc, options)
+    end
+  end
+end
+

data/lib/sitemap_generator/railtie.rb

@@ -0,0 +1,7 @@
+module SitemapGenerator
+  class Railtie < Rails::Railtie
+    rake_tasks do
+      load File.expand_path('../../../tasks/sitemap_generator_tasks.rake', __FILE__)
+    end
+  end
+end

data/lib/sitemap_generator/tasks.rb

@@ -0,0 +1 @@
+load File.expand_path(File.join(File.dirname(__FILE__), '../../tasks/sitemap_generator_tasks.rake'))

data/lib/sitemap_generator/templates.rb

@@ -0,0 +1,41 @@
+module SitemapGenerator
+  # Provide convenient access to template files.  E.g.
+  #
+  #   SitemapGenerator.templates.sitemap_index
+  #
+  # Lazy-load and cache for efficient access.
+  # Define an accessor method for each template file.
+  class Templates
+    FILES = {
+      :sitemap_sample =>  'sitemap.rb',
+    }
+
+    # Dynamically define accessors for each key defined in <tt>FILES</tt>
+    attr_accessor *FILES.keys
+    FILES.keys.each do |name|
+      eval <<-END
+        define_method(:#{name}) do
+          @#{name} ||= read_template(:#{name})
+        end
+      END
+    end
+
+    def initialize(root = SitemapGenerator.root)
+      @root = root
+    end
+
+    # Return the full path to a template.
+    #
+    # <tt>file</tt> template symbol e.g. <tt>:sitemap_sample</tt>
+    def template_path(template)
+      File.join(@root, 'templates', self.class::FILES[template])
+    end
+
+    protected
+
+    # Read the template file and return its contents.
+    def read_template(template)
+      File.read(template_path(template))
+    end
+  end
+end

data/lib/sitemap_generator/utilities.rb

@@ -0,0 +1,54 @@
+module SitemapGenerator
+  module Utilities
+    extend self
+
+    # Copy templates/sitemap.rb to config if not there yet.
+    def install_sitemap_rb(verbose=false)
+      if File.exist?(File.join(RAILS_ROOT, 'config/sitemap.rb'))
+        puts "already exists: config/sitemap.rb, file not copied" if verbose
+      else
+        FileUtils.cp(
+          SitemapGenerator.templates.template_path(:sitemap_sample),
+          File.join(RAILS_ROOT, 'config/sitemap.rb'))
+        puts "created: config/sitemap.rb" if verbose
+      end
+    end
+
+    # Remove config/sitemap.rb if exists.
+    def uninstall_sitemap_rb
+      if File.exist?(File.join(RAILS_ROOT, 'config/sitemap.rb'))
+        File.rm(File.join(RAILS_ROOT, 'config/sitemap.rb'))
+      end
+    end
+
+    # Clean sitemap files in output directory.
+    def clean_files
+      FileUtils.rm(Dir[File.join(RAILS_ROOT, 'public/sitemap*.xml.gz')])
+    end
+
+    # Returns whether this environment is using ActionPack
+    # version 3.0.0 or greater.
+    #
+    # @return [Boolean]
+    def self.rails3?
+      # The ActionPack module is always loaded automatically in Rails >= 3
+      return false unless defined?(ActionPack) && defined?(ActionPack::VERSION)
+
+      version =
+        if defined?(ActionPack::VERSION::MAJOR)
+          ActionPack::VERSION::MAJOR
+        else
+          # Rails 1.2
+          ActionPack::VERSION::Major
+        end
+
+      # 3.0.0.beta1 acts more like ActionPack 2
+      # for purposes of this method
+      # (checking whether block helpers require = or -).
+      # This extra check can be removed when beta2 is out.
+      version >= 3 &&
+        !(defined?(ActionPack::VERSION::TINY) &&
+          ActionPack::VERSION::TINY == "0.beta")
+    end
+  end
+end

data/rails/install.rb

@@ -0,0 +1,2 @@
+# Install hook code here
+SitemapGenerator::Utilities.install_sitemap_rb

data/rails/uninstall.rb

@@ -0,0 +1,2 @@
+# Uninstall hook code here
+SitemapGenerator::Utilities.uninstall_sitemap_rb

data/tasks/sitemap_generator_tasks.rake

@@ -0,0 +1,31 @@
+begin
+  require 'sitemap_generator'
+rescue LoadError, NameError
+  # Application should work without vlad
+end
+
+namespace :sitemap do
+  desc "Install a default config/sitemap.rb file"
+  task :install do
+    SitemapGenerator::Utilities.install_sitemap_rb(verbose)
+  end
+
+  desc "Delete all Sitemap files in public/ directory"
+  task :clean do
+    SitemapGenerator::Utilities.clean_files
+  end
+
+  desc "Create Sitemap XML files in public/ directory (rake -s for no output)"
+  task :refresh => ['sitemap:create'] do
+    SitemapGenerator::Sitemap.ping_search_engines
+  end
+
+  desc "Create Sitemap XML files (don't ping search engines)"
+  task 'refresh:no_ping' => ['sitemap:create']
+
+  task :create => [:environment] do
+    SitemapGenerator::Sitemap.verbose = verbose
+    SitemapGenerator::Sitemap.create
+  end
+end
+

data/templates/sitemap.rb

@@ -0,0 +1,42 @@
+# Set the host name for URL creation
+SitemapGenerator::Sitemap.default_host = "http://www.example.com"
+
+SitemapGenerator::Sitemap.add_links do |sitemap|
+  # Put links creation logic here.
+  #
+  # The root path '/' and sitemap index file are added automatically.
+  # Links are added to the Sitemap in the order they are specified.
+  #
+  # Usage: sitemap.add path, options
+  #        (default options are used if you don't specify)
+  #
+  # Defaults: :priority => 0.5, :changefreq => 'weekly',
+  #           :lastmod => Time.now, :host => default_host
+
+
+  # Examples:
+
+  # add '/articles'
+  sitemap.add articles_path, :priority => 0.7, :changefreq => 'daily'
+
+  # add all individual articles
+  Article.find(:all).each do |a|
+    sitemap.add article_path(a), :lastmod => a.updated_at
+  end
+
+  # add merchant path
+  sitemap.add '/purchase', :priority => 0.7, :host => "https://www.example.com"
+
+end
+
+# Including Sitemaps from Rails Engines.
+#
+# These Sitemaps should be almost identical to a regular Sitemap file except
+# they needn't define their own SitemapGenerator::Sitemap.default_host since
+# they will undoubtedly share the host name of the application they belong to.
+#
+# As an example, say we have a Rails Engine in vendor/plugins/cadability_client
+# We can include its Sitemap here as follows:
+#
+# file = File.join(Rails.root, 'vendor/plugins/cadability_client/config/sitemap.rb')
+# eval(open(file).read, binding, file)

metadata

@@ -0,0 +1,115 @@
+--- !ruby/object:Gem::Specification 
+name: airblade-sitemap_generator
+version: !ruby/object:Gem::Version 
+  hash: 27
+  prerelease: false
+  segments: 
+  - 0
+  - 3
+  - 4
+  version: 0.3.4
+platform: ruby
+authors: 
+- Adam Salter
+- Karl Varga
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2010-06-28 00:00:00 +01:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: rspec
+  prerelease: false
+  requirement: &id001 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 3
+        segments: 
+        - 0
+        version: "0"
+  type: :development
+  version_requirements: *id001
+- !ruby/object:Gem::Dependency 
+  name: nokogiri
+  prerelease: false
+  requirement: &id002 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 3
+        segments: 
+        - 0
+        version: "0"
+  type: :development
+  version_requirements: *id002
+description: A Rails 3-compatible gem/plugin to generate enterprise-class Sitemaps using a familiar Rails Routes-like DSL.  Sitemaps are readable by all search engines and adhere to the Sitemap protocol specification.  Automatically pings search engines to notify them of new sitemaps (including Google, Yahoo and Bing).  Provides rake tasks to easily manage your sitemaps.  Supports image sitemaps and handles millions of links.
+email: boss@airbladesoftware.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: 
+- README.md
+files: 
+- MIT-LICENSE
+- README.md
+- Rakefile
+- VERSION
+- lib/sitemap_generator.rb
+- lib/sitemap_generator/builder.rb
+- lib/sitemap_generator/builder/helper.rb
+- lib/sitemap_generator/builder/sitemap_file.rb
+- lib/sitemap_generator/builder/sitemap_index_file.rb
+- lib/sitemap_generator/interpreter.rb
+- lib/sitemap_generator/link.rb
+- lib/sitemap_generator/link_set.rb
+- lib/sitemap_generator/mapper.rb
+- lib/sitemap_generator/railtie.rb
+- lib/sitemap_generator/tasks.rb
+- lib/sitemap_generator/templates.rb
+- lib/sitemap_generator/utilities.rb
+- rails/install.rb
+- rails/uninstall.rb
+- tasks/sitemap_generator_tasks.rake
+- templates/sitemap.rb
+has_rdoc: true
+homepage: http://github.com/airblade/sitemap_generator
+licenses: []
+
+post_install_message: 
+rdoc_options: 
+- --charset=UTF-8
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.3.7
+signing_key: 
+specification_version: 3
+summary: Easily generate enterprise class Sitemaps for your Rails site using a familiar Rails Routes-like DSL
+test_files: []
+