apsoto-sitemap_generator 1.0.1.dev

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 [name of plugin creator]
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,227 @@
+SitemapGenerator
+================
+
+SitemapGenerator is a Rails gem that makes it easy to generate ['enterprise-class'][enterprise_class] Sitemaps readable by all search engines.  Generated Sitemaps adhere to the ['Sitemap protocol specification'][sitemap_protocol].  When you generate new Sitemaps, SitemapGenerator can automatically ping the major search engines (including Google, Yahoo and Bing) to notify them.  SitemapGenerator includes rake tasks to easily manage your sitemaps.
+
+Features
+-------
+
+- v0.2.6: ['Google Image Sitemap'][sitemap_images] support
+- v0.2.5: Rails 3 support (beta)
+
+- Adheres to the ['Sitemap protocol specification'][sitemap_protocol]
+- Handles millions of links
+- Automatic Gzip of Sitemap files
+- Automatic ping of search engines to notify them of new sitemaps: Google, Yahoo, Bing, Ask, SitemapWriter
+- Leaves your old sitemaps in place if a new one fails to generate
+- Allows you to set the hostname for the links in your Sitemap
+
+Foreword
+-------
+
+Unfortunately, Adam Salter passed away in 2009.  Those who knew him know what an amazing guy he was, and what an excellent Rails programmer he was.  His passing is a great loss to the Rails community.
+
+[Karl Varga](http://github.com/kjvarga) has taken over development of SitemapGenerator.  The canonical repository is [http://github.com/kjvarga/sitemap_generator][canonical_repo]
+
+Installation
+=======
+
+**Rails 3:**
+
+1. Add the gem to your <tt>Gemspec</tt>
+
+    <code>gem 'sitemap_generator'</code>
+
+2. `$ rake sitemap:install`
+
+**Rails 2.x: As a gem**
+
+1. Add the gem as a dependency in your <tt>config/environment.rb</tt>
+
+    <code>config.gem 'sitemap_generator', :lib => false</code>
+
+2. `$ rake gems:install`
+
+3. Add the following to your <tt>RAILS_ROOT/Rakefile</tt>
+
+    <pre>begin
+      require 'sitemap_generator/tasks'
+    rescue Exception => e
+      puts "Warning, couldn't load gem tasks: #{e.message}! Skipping..."
+    end</pre>
+
+4. `$ rake sitemap:install`
+
+**Rails 2.x: As a plugin**
+
+1. <code>$ ./script/plugin install git://github.com/kjvarga/sitemap_generator.git</code>
+
+----
+
+Installation creates a <tt>config/sitemap.rb</tt> file which will contain your logic for generating the Sitemap files.  If you want to create this file manually run <code>rake sitemap:install</code>.
+
+You can run <code>rake sitemap:refresh</code> as needed to create Sitemap files. This will also ping these ['major search engines'][sitemap_engines]: Google, Yahoo, Bing, Ask, SitemapWriter.  If you want to disable all non-essential output run the rake task with <code>rake -s sitemap:refresh</code>.
+
+To keep your Sitemaps up-to-date, setup a cron job.  Pass the <tt>-s</tt> option to the rake task to silence all but the most important output.  If you're using Whenever, then your schedule would look something like:
+
+    # config/schedule.rb
+    every 1.day, :at => '5:00 am' do
+      rake "-s sitemap:refresh"
+    end
+
+Optionally, you can add the following to your <code>public/robots.txt</code> file, so that robots can find the sitemap file:
+
+    Sitemap: <hostname>/sitemap_index.xml.gz
+
+The Sitemap URL in the robots file should be the complete URL to the Sitemap Index, such as <tt>http://www.example.org/sitemap_index.xml.gz</tt>
+
+
+Example 'config/sitemap.rb'
+==========
+
+    # Set the host name for URL creation
+    SitemapGenerator::Sitemap.default_host = "http://www.example.com"
+
+    SitemapGenerator::Sitemap.add_links do |sitemap|
+      # Put links creation logic here.
+      #
+      # The Root Path ('/') and Sitemap Index file are added automatically.
+      # Links are added to the Sitemap output in the order they are specified.
+      #
+      # Usage: sitemap.add path, options
+      #        (default options are used if you don't specify them)
+      #
+      # Defaults: :priority => 0.5, :changefreq => 'weekly',
+      #           :lastmod => Time.now, :host => default_host
+
+
+      # Examples:
+
+      # add '/articles'
+      sitemap.add articles_path, :priority => 0.7, :changefreq => 'daily'
+
+      # add all individual articles
+      Article.find(:all).each do |a|
+        sitemap.add article_path(a), :lastmod => a.updated_at
+      end
+
+      # add merchant path
+      sitemap.add '/purchase', :priority => 0.7, :host => "https://www.example.com"
+
+      # add all individual news with images
+      News.all.each do |n|
+        sitemap.add news_path(n), :lastmod => n.updated_at, :images=>n.images.collect{ |r| :loc=>r.image.url, :title=>r.image.name }
+      end
+
+    end
+
+    # Including Sitemaps from Rails Engines.
+    #
+    # These Sitemaps should be almost identical to a regular Sitemap file except
+    # they needn't define their own SitemapGenerator::Sitemap.default_host since
+    # they will undoubtedly share the host name of the application they belong to.
+    #
+    # As an example, say we have a Rails Engine in vendor/plugins/cadability_client
+    # We can include its Sitemap here as follows:
+    #
+    file = File.join(Rails.root, 'vendor/plugins/cadability_client/config/sitemap.rb')
+    eval(open(file).read, binding, file)
+
+Raison d'être
+-------
+
+Most of the Sitemap plugins out there seem to try to recreate the Sitemap links by iterating the Rails routes. In some cases this is possible, but for a great deal of cases it isn't.
+
+a) There are probably quite a few routes in your routes file that don't need inclusion in the Sitemap. (AJAX routes I'm looking at you.)
+
+and
+
+b) How would you infer the correct series of links for the following route?
+
+    map.zipcode 'location/:state/:city/:zipcode', :controller => 'zipcode', :action => 'index'
+
+Don't tell me it's trivial, because it isn't. It just looks trivial.
+
+So my idea is to have another file similar to 'routes.rb' called 'sitemap.rb', where you can define what goes into the Sitemap.
+
+Here's my solution:
+
+    Zipcode.find(:all, :include => :city).each do |z|
+      sitemap.add zipcode_path(:state => z.city.state, :city => z.city, :zipcode => z)
+    end
+
+Easy hey?
+
+Other Sitemap settings for the link, like `lastmod`, `priority`, `changefreq` and `host` are entered automatically, although you can override them if you need to.
+
+Compatibility
+=======
+
+Tested and working on:
+
+- **Rails** 3.0.0, sitemap_generator version >= 0.2.5
+- **Rails** 1.x - 2.3.5
+- **Ruby** 1.8.6, 1.8.7, 1.9.1
+
+Notes
+=======
+
+1) For large sitemaps it may be useful to split your generation into batches to avoid running out of memory. E.g.:
+
+    # add movies
+    Movie.find_in_batches(:batch_size => 1000) do |movies|
+      movies.each do |movie|
+        sitemap.add "/movies/show/#{movie.to_param}", :lastmod => movie.updated_at, :changefreq => 'weekly'
+      end
+    end
+
+2) New Capistrano deploys will remove your Sitemap files, unless you run `rake sitemap:refresh`. The way around this is to create a cap task:
+
+    after "deploy:update_code", "deploy:copy_old_sitemap"
+
+    namespace :deploy do
+      task :copy_old_sitemap do
+          run "if [ -e #{previous_release}/public/sitemap_index.xml.gz ]; then cp #{previous_release}/public/sitemap* #{current_release}/public/; fi"
+      end
+    end
+
+3) If generation of your sitemap fails for some reason, the old sitemap will remain in public/.  This ensures that robots will always find a valid sitemap.  Running silently (`rake -s sitemap:refresh`) and with email forwarding setup you'll only get an email if your sitemap fails to build, and no notification when everything is fine - which will be most of the time.
+
+Known Bugs
+========
+
+- There's no check on the size of a URL which [isn't supposed to exceed 2,048 bytes][sitemaps_xml].
+- Currently only supports one Sitemap Index file, which can contain 50,000 Sitemap files which can each contain 50,000 urls, so it _only_ supports up to 2,500,000,000 (2.5 billion) urls. I personally have no need of support for more urls, but plugin could be improved to support this.
+
+Wishlist & Coming Soon
+========
+
+- Support for generating sitemaps for sites with multiple domains.  Sitemaps are generated into subdirectories and we use a Rack middleware to rewrite requests for sitemaps to the correct subdirectory based on the request host.
+- I want to refactor the code because it has grown a lot.  Part of this refactoring will include implementing some more checks to make sure we adhere to standards as well as making sure that the sitemaps are being generated as efficiently as possible.
+
+I'd like to simplify adding links to a sitemap.  Right now it's all or nothing.  I'd like to break it up so you can add batches.
+- Auto coverage testing.  Generate a report of broken URLs by checking the status codes of each page in the sitemap.
+
+Thanks (in no particular order)
+========
+
+- [Alexadre Bini](http://github.com/alexandrebini) for image sitemaps
+- [Dan Pickett](http://github.com/dpickett)
+- [Rob Biedenharn](http://github.com/rab)
+- [Richie Vos](http://github.com/jerryvos)
+- [Adrian Mugnolo](http://github.com/xymbol)
+- [Jason Weathered](http://github.com/jasoncodes)
+- [Andy Stewart](http://github.com/airblade)
+
+Copyright (c) 2009 Karl Varga released under the MIT license
+
+[canonical_repo]:http://github.com/kjvarga/sitemap_generator
+[enterprise_class]:https://twitter.com/dhh/status/1631034662 "I use enterprise in the same sense the Phusion guys do - i.e. Enterprise Ruby. Please don't look down on my use of the word 'enterprise' to represent being a cut above. It doesn't mean you ever have to work for a company the size of IBM. Or constantly fight inertia, writing crappy software, adhering to change management practices and spending hours in meetings... Not that there's anything wrong with that - Wait, what?"
+[sitemap_engines]:http://en.wikipedia.org/wiki/Sitemap_index "http://en.wikipedia.org/wiki/Sitemap_index"
+[sitemaps_org]:http://www.sitemaps.org/protocol.php "http://www.sitemaps.org/protocol.php"
+[sitemaps_xml]:http://www.sitemaps.org/protocol.php#xmlTagDefinitions "XML Tag Definitions"
+[sitemap_generator_usage]:http://wiki.github.com/adamsalter/sitemap_generator/sitemapgenerator-usage "http://wiki.github.com/adamsalter/sitemap_generator/sitemapgenerator-usage"
+[boost_juice]:http://www.boostjuice.com.au/ "Mmmm, sweet, sweet Boost Juice."
+[cb]:http://codebright.net "http://codebright.net"
+[sitemap_images]:http://www.google.com/support/webmasters/bin/answer.py?answer=178636
+[sitemap_protocol]:http://sitemaps.org/protocol.php

data/Rakefile

@@ -0,0 +1,123 @@
+require 'rake'
+require 'rake/rdoctask'
+require 'rubygems'
+gem 'rspec', '1.3.0'
+require 'spec/rake/spectask'
+gem 'nokogiri'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    gem.name = "apsoto-sitemap_generator"
+    gem.summary = %Q{Easily generate enterprise class Sitemaps for your Rails site using a familiar Rails Routes-like DSL}
+    gem.description = %Q{SitemapGenerator is a Rails gem that makes it easy to generate enterprise-class Sitemaps readable by all search engines.  Generated Sitemaps adhere to the Sitemap protocol specification.  When you generate new Sitemaps, SitemapGenerator can automatically ping the major search engines (including Google, Yahoo and Bing) to notify them.  SitemapGenerator includes rake tasks to easily manage your sitemaps.}
+    gem.email = "apsoto@gmail.com"
+    gem.homepage = "http://github.com/apsoto/sitemap_generator"
+    gem.authors = ["Alex Soto", "Karl Varga", "Adam Salter"]
+    gem.files =  FileList["[A-Z]*", "{bin,lib,rails,templates,tasks}/**/*"]
+    gem.test_files = []
+    gem.add_development_dependency "rspec"
+    gem.add_development_dependency "nokogiri"
+    gem.add_development_dependency "sqlite3-ruby"
+  end
+  Jeweler::GemcutterTasks.new
+rescue LoadError
+  puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
+end
+
+#
+# Helper methods
+#
+module Helpers
+  extend self
+
+  # Return a full local path to path fragment <tt>path</tt>
+  def local_path(path)
+    File.join(File.dirname(__FILE__), path)
+  end
+
+  # Copy all of the local files into <tt>path</tt> after completely cleaning it
+  def prepare_path(path)
+    rm_rf path
+    mkdir_p path
+    cp_r(FileList["[A-Z]*", "{bin,lib,rails,templates,tasks}"], path)
+  end
+end
+
+#
+# Tasks
+#
+task :default => :test
+
+namespace :test do
+  #desc "Test as a gem, plugin and Rails 3 gem"
+  #task :all => ['test:gem', 'test:plugin']
+
+  task :gem => ['test:prepare:gem', 'multi_spec']
+  task :plugin => ['test:prepare:plugin', 'multi_spec']
+  task :rails3 => ['test:prepare:rails3', 'multi_spec']
+
+  task :multi_spec do
+    Rake::Task['spec'].invoke
+    Rake::Task['spec'].reenable
+  end
+
+  namespace :prepare do
+    task :gem do
+      ENV["SITEMAP_RAILS"] = 'gem'
+      Helpers.prepare_path(Helpers.local_path('spec/mock_app_gem/vendor/gems/sitemap_generator-1.2.3'))
+      rm_rf(Helpers.local_path('spec/mock_app_gem/public/sitemap*'))
+    end
+
+    task :plugin do
+      ENV["SITEMAP_RAILS"] = 'plugin'
+      Helpers.prepare_path(Helpers.local_path('spec/mock_app_plugin/vendor/plugins/sitemap_generator-1.2.3'))
+      rm_rf(Helpers.local_path('spec/mock_app_plugin/public/sitemap*'))
+    end
+
+    task :rails3 do
+      ENV["SITEMAP_RAILS"] = 'rails3'
+      rm_rf(Helpers.local_path('spec/mock_rails3_gem/public/sitemap*'))
+    end
+  end
+end
+
+desc "Run tests as a gem install"
+task :test => ['test:gem']
+
+Spec::Rake::SpecTask.new(:spec) do |spec|
+  spec.libs << 'lib' << 'spec'
+  spec.spec_files = FileList['spec/**/*_spec.rb']
+end
+task :spec => :check_dependencies
+
+Spec::Rake::SpecTask.new(:rcov) do |spec|
+  spec.libs << 'lib' << 'spec'
+  spec.pattern = 'spec/**/*_spec.rb'
+  spec.rcov = true
+end
+
+desc 'Generate documentation'
+Rake::RDocTask.new(:rdoc) do |rdoc|
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title    = 'SitemapGenerator'
+  rdoc.options << '--line-numbers' << '--inline-source'
+  rdoc.rdoc_files.include('README.md')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+namespace :release do
+
+  desc "Release a new patch version"
+  task :patch do
+    Rake::Task['version:bump:patch'].invoke
+    Rake::Task['release:current'].invoke
+  end
+
+  desc "Release the current version (e.g. after a version bump).  This rebuilds the gemspec, pushes the updated code, tags it and releases to RubyGems"
+  task :current do
+    Rake::Task['github:release'].invoke
+    Rake::Task['git:release'].invoke
+    Rake::Task['gemcutter:release'].invoke
+  end
+end

data/VERSION

@@ -0,0 +1 @@
+1.0.1.dev

data/lib/sitemap_generator/builder/helper.rb

@@ -0,0 +1,10 @@
+module SitemapGenerator
+  module Builder
+    module Helper
+
+      def w3c_date(date)
+         date.utc.strftime("%Y-%m-%dT%H:%M:%S+00:00")
+      end
+    end
+  end
+end

data/lib/sitemap_generator/builder/sitemap_file.rb

@@ -0,0 +1,155 @@
+require 'sitemap_generator/builder/helper'
+require 'builder'
+require 'zlib'
+
+module SitemapGenerator
+  module Builder
+    class SitemapFile
+      include SitemapGenerator::Builder::Helper
+
+      attr_accessor :sitemap_path, :public_path, :filesize, :link_count, :hostname
+
+      # <tt>public_path</tt> full path of the directory to write sitemaps in.
+      #   Usually your Rails <tt>public/</tt> directory.
+      #
+      # <tt>sitemap_path</tt> relative path including filename of the sitemap
+      #   file relative to <tt>public_path</tt>
+      #
+      # <tt>hostname</tt> hostname including protocol to use in all links
+      #   e.g. http://en.google.ca
+      def initialize(public_path, sitemap_path, hostname)
+        self.sitemap_path = sitemap_path
+        self.public_path = public_path
+        self.hostname = hostname
+        self.link_count = 0
+
+        @xml_content       = ''     # XML urlset content
+        @xml_wrapper_start = <<-HTML
+          <?xml version="1.0" encoding="UTF-8"?>
+            <urlset
+              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+              xmlns:image="http://www.google.com/schemas/sitemap-image/1.1"
+              xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9
+                http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd"
+              xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
+              xmlns:video="http://www.google.com/schemas/sitemap-video/1.1"
+            >
+        HTML
+        @xml_wrapper_start.gsub!(/\s+/, ' ').gsub!(/ *> */, '>').strip!
+        @xml_wrapper_end   = %q[</urlset>]
+        self.filesize = bytesize(@xml_wrapper_start) + bytesize(@xml_wrapper_end)
+      end
+
+      def lastmod
+        File.mtime(self.full_path) rescue nil
+      end
+
+      def empty?
+        self.link_count == 0
+      end
+
+      def full_url
+        URI.join(self.hostname, self.sitemap_path).to_s
+      end
+
+      def full_path
+        @full_path ||= File.join(self.public_path, self.sitemap_path)
+      end
+
+      # Return a boolean indicating whether the sitemap file can fit another link
+      # of <tt>bytes</tt> bytes in size.
+      def file_can_fit?(bytes)
+        (self.filesize + bytes) < SitemapGenerator::MAX_SITEMAP_FILESIZE && self.link_count < SitemapGenerator::MAX_SITEMAP_LINKS
+      end
+
+      # Add a link to the sitemap file and return a boolean indicating whether the
+      # link was added.
+      #
+      # If a link cannot be added, the file is too large or the link limit has been reached.
+      def add_link(link)
+        xml = build_xml(::Builder::XmlMarkup.new, link)
+        unless file_can_fit?(bytesize(xml))
+          self.finalize!
+          return false
+        end
+
+        @xml_content << xml
+        self.filesize += bytesize(xml)
+        self.link_count += 1
+        true
+      end
+      alias_method :<<, :add_link
+
+      # Return XML as a String
+      def build_xml(builder, link)
+        builder.url do
+          builder.loc        link[:loc]
+          builder.lastmod    w3c_date(link[:lastmod])   if link[:lastmod]
+          builder.changefreq link[:changefreq]          if link[:changefreq]
+          builder.priority   link[:priority]            if link[:priority]
+
+          unless link[:images].blank?
+            link[:images].each do |image|
+              builder.image:image do
+                builder.image :loc, image[:loc]
+                builder.image :caption, image[:caption]             if image[:caption]
+                builder.image :geo_location, image[:geo_location]   if image[:geo_location]
+                builder.image :title, image[:title]                 if image[:title]
+                builder.image :license, image[:license]             if image[:license]
+              end
+            end
+          end
+
+          unless link[:video].blank?
+            video = link[:video]
+            builder.video :video do
+              # required elements
+              builder.video :thumbnail_loc, video[:thumbnail_loc]
+              builder.video :title, video[:title]
+              builder.video :description, video[:description]
+
+              builder.video :content_loc, video[:content_loc]           if video[:content_loc]
+              if video[:player_loc]
+                builder.video :player_loc, video[:player_loc], :allow_embed => (video[:allow_embed] ? 'yes' : 'no'), :autoplay => video[:autoplay]
+              end
+
+              builder.video :rating, video[:rating]                     if video[:rating]
+              builder.video :view_count, video[:view_count]             if video[:view_count]
+              builder.video :publication_date, video[:publication_date] if video[:publication_date]
+              builder.video :expiration_date, video[:expiration_date]   if video[:expiration_date]
+              builder.video :duration, video[:duration]                 if video[:duration]
+              builder.video :family_friendly, (video[:family_friendly] ? 'yes' : 'no')  if video[:family_friendly]
+              builder.video :duration, video[:duration]                 if video[:duration]
+              video[:tags].each {|tag| builder.video :tag, tag }        if video[:tags]
+              video[:categories].each {|category| builder.video :category, category} if video[:categories]
+            end
+          end
+        end
+        builder << ''
+      end
+
+      # Insert the content into the XML "wrapper" and write and close the file.
+      #
+      # All the xml content in the instance is cleared, but attributes like
+      # <tt>filesize</tt> are still available.
+      def finalize!
+        return if self.frozen?
+
+        open(self.full_path, 'wb') do |file|
+          gz = Zlib::GzipWriter.new(file)
+          gz.write @xml_wrapper_start
+          gz.write @xml_content
+          gz.write @xml_wrapper_end
+          gz.close
+        end
+        @xml_content = @xml_wrapper_start = @xml_wrapper_end = ''
+        self.freeze
+      end
+
+      # Return the bytesize length of the string
+      def bytesize(string)
+        string.respond_to?(:bytesize) ? string.bytesize : string.length
+      end
+    end
+  end
+end

data/lib/sitemap_generator/builder/sitemap_index_file.rb

@@ -0,0 +1,33 @@
+module SitemapGenerator
+  module Builder
+    class SitemapIndexFile < SitemapFile
+
+      def initialize(*args)
+        super(*args)
+
+        @xml_content = '' # XML urlset content
+        @xml_wrapper_start = <<-HTML
+          <?xml version="1.0" encoding="UTF-8"?>
+            <sitemapindex
+              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+              xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9
+                http://www.sitemaps.org/schemas/sitemap/0.9/siteindex.xsd"
+              xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
+            >
+        HTML
+        @xml_wrapper_start.gsub!(/\s+/, ' ').gsub!(/ *> */, '>').strip!
+        @xml_wrapper_end   = %q[</sitemapindex>]
+        self.filesize = bytesize(@xml_wrapper_start) + bytesize(@xml_wrapper_end)
+      end
+
+      # Return XML as a String
+      def build_xml(builder, link)
+        builder.sitemap do
+          builder.loc        link[:loc]
+          builder.lastmod    w3c_date(link[:lastmod])   if link[:lastmod]
+        end
+        builder << ''
+      end
+    end
+  end
+end

data/lib/sitemap_generator/builder.rb

@@ -0,0 +1,9 @@
+require 'sitemap_generator/builder/helper'
+require 'sitemap_generator/builder/sitemap_file'
+require 'sitemap_generator/builder/sitemap_index_file'
+
+module SitemapGenerator
+  module Builder
+
+  end
+end

data/lib/sitemap_generator/interpreter.rb

@@ -0,0 +1,28 @@
+module SitemapGenerator
+
+  # Evaluate a sitemap config file within the context of a class that includes the
+  # Rails URL helpers.
+  class Interpreter
+
+    if SitemapGenerator::Utilities.rails3?
+      include ::Rails.application.routes.url_helpers
+    else
+      require 'action_controller'
+      include ActionController::UrlWriter
+    end
+
+    def initialize(sitemap_config_file=nil)
+      sitemap_config_file ||= File.join(::Rails.root, 'config/sitemap.rb')
+      eval(open(sitemap_config_file).read)
+    end
+
+    # KJV do we need this?  We should be using path_* helpers.
+    # def self.default_url_options(options = nil)
+    #   { :host => SitemapGenerator::Sitemap.default_host }
+    # end
+
+    def self.run
+      new
+    end
+  end
+end

data/lib/sitemap_generator/link.rb

@@ -0,0 +1,37 @@
+module SitemapGenerator
+  module Link
+    extend self
+
+    # Return a Hash of options suitable to pass to a SitemapGenerator::Builder::SitemapFile instance.
+    def generate(path, options = {})
+      if path.is_a?(SitemapGenerator::Builder::SitemapFile)
+        options.reverse_merge!(:host => path.hostname, :lastmod => path.lastmod)
+        path = path.sitemap_path
+      end
+
+      options.assert_valid_keys(:priority, :changefreq, :lastmod, :host, :images, :video)
+      options.reverse_merge!(:priority => 0.5, :changefreq => 'weekly', :lastmod => Time.now, :host => Sitemap.default_host, :images => [])
+      {
+        :path => path,
+        :priority => options[:priority],
+        :changefreq => options[:changefreq],
+        :lastmod => options[:lastmod],
+        :host => options[:host],
+        :loc => URI.join(options[:host], path).to_s,
+        :images => prepare_images(options[:images], options[:host]),
+        :video => options[:video]
+      }
+    end
+
+    # Return an Array of image option Hashes suitable to be parsed by SitemapGenerator::Builder::SitemapFile
+    def prepare_images(images, host)
+      images.delete_if { |key,value| key[:loc] == nil }
+      images.each do |r|
+        r.assert_valid_keys(:loc, :caption, :geo_location, :title, :license)
+        r[:loc] = URI.join(host, r[:loc]).to_s
+      end
+      images[0..(SitemapGenerator::MAX_SITEMAP_IMAGES-1)]
+    end
+  end
+end
+

data/lib/sitemap_generator/link_set.rb

@@ -0,0 +1,174 @@
+require 'builder'
+require 'action_view'
+
+# A LinkSet provisions a bunch of links to sitemap files.  It also writes the index file
+# which lists all the sitemap files written.
+module SitemapGenerator
+  class LinkSet
+    include ActionView::Helpers::NumberHelper  # for number_with_delimiter
+
+    attr_accessor :default_host, :public_path, :sitemaps_path
+    attr_accessor :sitemap, :sitemaps, :sitemap_index
+    attr_accessor :verbose, :yahoo_app_id
+
+    # Evaluate the sitemap config file and write all sitemaps.
+    #
+    # This should be refactored so that we can have multiple instances
+    # of LinkSet.
+    def create
+      require 'sitemap_generator/interpreter'
+
+      self.public_path = File.join(::Rails.root, 'public/') if self.public_path.nil?
+
+      start_time = Time.now
+      SitemapGenerator::Interpreter.run
+      finalize!
+      end_time = Time.now
+
+      puts "\nSitemap stats: #{number_with_delimiter(self.link_count)} links / #{self.sitemaps.size} files / " + ("%dm%02ds" % (end_time - start_time).divmod(60)) if verbose
+    end
+
+    # <tt>public_path</tt> (optional) full path to the directory to write sitemaps in.
+    #   Defaults to your Rails <tt>public/</tt> directory.
+    #
+    # <tt>sitemaps_path</tt> (optional) path fragment within public to write sitemaps
+    #   to e.g. 'en/'.  Sitemaps are written to <tt>public_path</tt> + <tt>sitemaps_path</tt>
+    #
+    # <tt>default_host</tt> hostname including protocol to use in all sitemap links
+    #   e.g. http://en.google.ca
+    def initialize(public_path = nil, sitemaps_path = nil, default_host = nil)
+      self.default_host = default_host
+      self.public_path = public_path
+      self.sitemaps_path = sitemaps_path
+
+      # Completed sitemaps
+      self.sitemaps = []
+    end
+
+    def link_count
+      self.sitemaps.inject(0) { |link_count_sum, sitemap| link_count_sum + sitemap.link_count }
+    end
+
+    # Called within the user's eval'ed sitemap config file.  Add links to sitemap files
+    # passing a block.
+    #
+    # TODO: Refactor.  The call chain is confusing and convoluted here.
+    def add_links
+      raise ArgumentError, "Default hostname not set" if default_host.blank?
+
+      # I'd rather have these calls in <tt>create</tt> but we have to wait
+      # for <tt>default_host</tt> to be set by the user's sitemap config
+      new_sitemap
+      add_default_links
+
+      yield Mapper.new(self)
+    end
+
+    # Called from Mapper.
+    #
+    # Add a link to the current sitemap.
+    def add_link(link)
+      unless self.sitemap << link
+        new_sitemap
+        self.sitemap << link
+      end
+    end
+
+    # Add the current sitemap to the <tt>sitemaps</tt> Array and
+    # start a new sitemap.
+    #
+    # If the current sitemap is nil or empty it is not added.
+    def new_sitemap
+      unless self.sitemap_index
+        self.sitemap_index = SitemapGenerator::Builder::SitemapIndexFile.new(public_path, sitemap_index_path, default_host)
+      end
+
+      unless self.sitemap
+        self.sitemap = SitemapGenerator::Builder::SitemapFile.new(public_path, new_sitemap_path, default_host)
+      end
+
+      # Mark the sitemap as complete and add it to the sitemap index
+      unless self.sitemap.empty?
+        self.sitemap.finalize!
+        self.sitemap_index << Link.generate(self.sitemap)
+        self.sitemaps << self.sitemap
+        show_progress(self.sitemap) if verbose
+
+        self.sitemap = SitemapGenerator::Builder::SitemapFile.new(public_path, new_sitemap_path, default_host)
+      end
+    end
+
+    # Report progress line.
+    def show_progress(sitemap)
+      uncompressed_size = number_to_human_size(sitemap.filesize)
+      compressed_size =   number_to_human_size(File.size?(sitemap.full_path))
+      puts "+ #{sitemap.sitemap_path}   #{sitemap.link_count} links / #{uncompressed_size} / #{compressed_size} gzipped"
+    end
+
+    # Finalize all sitemap files
+    def finalize!
+      new_sitemap
+      self.sitemap_index.finalize!
+    end
+
+    # Ping search engines.
+    #
+    # @see http://en.wikipedia.org/wiki/Sitemap_index
+    def ping_search_engines
+      require 'open-uri'
+
+      sitemap_index_url = CGI.escape(self.sitemap_index.full_url)
+      search_engines = {
+        :google         => "http://www.google.com/webmasters/sitemaps/ping?sitemap=#{sitemap_index_url}",
+        :yahoo          => "http://search.yahooapis.com/SiteExplorerService/V1/ping?sitemap=#{sitemap_index_url}&appid=#{yahoo_app_id}",
+        :ask            => "http://submissions.ask.com/ping?sitemap=#{sitemap_index_url}",
+        :bing           => "http://www.bing.com/webmaster/ping.aspx?siteMap=#{sitemap_index_url}",
+        :sitemap_writer => "http://www.sitemapwriter.com/notify.php?crawler=all&url=#{sitemap_index_url}"
+      }
+
+      puts "\n" if verbose
+      search_engines.each do |engine, link|
+        next if engine == :yahoo && !self.yahoo_app_id
+        begin
+          open(link)
+          puts "Successful ping of #{engine.to_s.titleize}" if verbose
+        rescue Timeout::Error, StandardError => e
+          puts "Ping failed for #{engine.to_s.titleize}: #{e.inspect} (URL #{link})" if verbose
+        end
+      end
+
+      if !self.yahoo_app_id && verbose
+        puts "\n"
+        puts <<-END.gsub(/^\s+/, '')
+          To ping Yahoo you require a Yahoo AppID.  Add it to your config/sitemap.rb with:
+
+          SitemapGenerator::Sitemap.yahoo_app_id = "my_app_id"
+
+          For more information see http://developer.yahoo.com/search/siteexplorer/V1/updateNotification.html
+        END
+      end
+    end
+
+    protected
+
+    def add_default_links
+      self.sitemap << Link.generate('/', :lastmod => Time.now, :changefreq => 'always', :priority => 1.0)
+      self.sitemap << Link.generate(self.sitemap_index, :lastmod => Time.now, :changefreq => 'always', :priority => 1.0)
+    end
+
+    # Return the current sitemap filename with index.
+    #
+    # The index depends on the length of the <tt>sitemaps</tt> array.
+    def new_sitemap_path
+      File.join(self.sitemaps_path || '', "sitemap#{self.sitemaps.length + 1}.xml.gz")
+    end
+
+    # Return the current sitemap index filename.
+    #
+    # At the moment we only support one index file which can link to
+    # up to 50,000 sitemap files.
+    def sitemap_index_path
+      File.join(self.sitemaps_path || '', 'sitemap_index.xml.gz')
+    end
+  end
+end

data/lib/sitemap_generator/mapper.rb

@@ -0,0 +1,16 @@
+module SitemapGenerator
+  # Generator instances are used to build links.
+  # The object passed to the add_links block in config/sitemap.rb is a Generator instance.
+  class Mapper
+    attr_accessor :set
+
+    def initialize(set)
+      @set = set
+    end
+
+    def add(loc, options = {})
+      set.add_link Link.generate(loc, options)
+    end
+  end
+end
+

data/lib/sitemap_generator/railtie.rb

@@ -0,0 +1,7 @@
+module SitemapGenerator
+  class Railtie < Rails::Railtie
+    rake_tasks do
+      load File.expand_path('../../../tasks/sitemap_generator_tasks.rake', __FILE__)
+    end
+  end
+end

data/lib/sitemap_generator/tasks.rb

@@ -0,0 +1 @@
+load File.expand_path(File.join(File.dirname(__FILE__), '../../tasks/sitemap_generator_tasks.rake'))

data/lib/sitemap_generator/templates.rb

@@ -0,0 +1,41 @@
+module SitemapGenerator
+  # Provide convenient access to template files.  E.g.
+  #
+  #   SitemapGenerator.templates.sitemap_index
+  #
+  # Lazy-load and cache for efficient access.
+  # Define an accessor method for each template file.
+  class Templates
+    FILES = {
+      :sitemap_sample =>  'sitemap.rb',
+    }
+
+    # Dynamically define accessors for each key defined in <tt>FILES</tt>
+    attr_accessor *FILES.keys
+    FILES.keys.each do |name|
+      eval <<-END
+        define_method(:#{name}) do
+          @#{name} ||= read_template(:#{name})
+        end
+      END
+    end
+
+    def initialize(root = SitemapGenerator.root)
+      @root = root
+    end
+
+    # Return the full path to a template.
+    #
+    # <tt>file</tt> template symbol e.g. <tt>:sitemap_sample</tt>
+    def template_path(template)
+      File.join(@root, 'templates', self.class::FILES[template])
+    end
+
+    protected
+
+    # Read the template file and return its contents.
+    def read_template(template)
+      File.read(template_path(template))
+    end
+  end
+end

data/lib/sitemap_generator/utilities.rb

@@ -0,0 +1,36 @@
+module SitemapGenerator
+  module Utilities
+    extend self
+
+    # Copy templates/sitemap.rb to config if not there yet.
+    def install_sitemap_rb(verbose=false)
+      if File.exist?(File.join(RAILS_ROOT, 'config/sitemap.rb'))
+        puts "already exists: config/sitemap.rb, file not copied" if verbose
+      else
+        FileUtils.cp(
+          SitemapGenerator.templates.template_path(:sitemap_sample),
+          File.join(RAILS_ROOT, 'config/sitemap.rb'))
+        puts "created: config/sitemap.rb" if verbose
+      end
+    end
+
+    # Remove config/sitemap.rb if exists.
+    def uninstall_sitemap_rb
+      if File.exist?(File.join(RAILS_ROOT, 'config/sitemap.rb'))
+        File.rm(File.join(RAILS_ROOT, 'config/sitemap.rb'))
+      end
+    end
+
+    # Clean sitemap files in output directory.
+    def clean_files
+      FileUtils.rm(Dir[File.join(RAILS_ROOT, 'public/sitemap*.xml.gz')])
+    end
+
+    # Returns a boolean indicating whether this environment is Rails 3
+    #
+    # @return [Boolean]
+    def self.rails3?
+      Rails.version.to_f >= 3
+    end
+  end
+end

data/lib/sitemap_generator.rb

@@ -0,0 +1,28 @@
+require 'sitemap_generator/builder'
+require 'sitemap_generator/mapper'
+require 'sitemap_generator/link'
+require 'sitemap_generator/link_set'
+require 'sitemap_generator/templates'
+require 'sitemap_generator/utilities'
+require 'sitemap_generator/railtie' if SitemapGenerator::Utilities.rails3?
+
+require 'active_support/core_ext/numeric'
+
+module SitemapGenerator
+  silence_warnings do
+    VERSION = File.read(File.dirname(__FILE__) + "/../VERSION").strip
+    MAX_SITEMAP_FILES    = 50_000        # max sitemap links per index file
+    MAX_SITEMAP_LINKS    = 50_000        # max links per sitemap
+    MAX_SITEMAP_IMAGES   = 1_000         # max images per url
+    MAX_SITEMAP_FILESIZE = 10.megabytes  # bytes
+
+    Sitemap = LinkSet.new
+  end
+
+  class << self
+    attr_accessor :root, :templates
+  end
+
+  self.root = File.expand_path(File.join(File.dirname(__FILE__), '../'))
+  self.templates = SitemapGenerator::Templates.new(self.root)
+end

data/rails/install.rb

@@ -0,0 +1,2 @@
+# Install hook code here
+SitemapGenerator::Utilities.install_sitemap_rb

data/rails/uninstall.rb

@@ -0,0 +1,2 @@
+# Uninstall hook code here
+SitemapGenerator::Utilities.uninstall_sitemap_rb

data/tasks/sitemap_generator_tasks.rake

@@ -0,0 +1,39 @@
+environment = begin
+
+  # Try to require the library.  If we are installed as a gem, this should work.
+  # We don't need to load the environment.
+  require 'sitemap_generator'
+  []
+
+rescue LoadError
+
+  # We must be installed as a plugin.  Make sure the environment is loaded
+  # when running all rake tasks.
+  [:environment]
+
+end
+
+namespace :sitemap do
+  desc "Install a default config/sitemap.rb file"
+  task :install => environment do
+    SitemapGenerator::Utilities.install_sitemap_rb(verbose)
+  end
+
+  desc "Delete all Sitemap files in public/ directory"
+  task :clean => environment do
+    SitemapGenerator::Utilities.clean_files
+  end
+
+  desc "Create Sitemap XML files in public/ directory (rake -s for no output)"
+  task :refresh => ['sitemap:create'] do
+    SitemapGenerator::Sitemap.ping_search_engines
+  end
+
+  desc "Create Sitemap XML files (don't ping search engines)"
+  task 'refresh:no_ping' => ['sitemap:create']
+
+  task :create => [:environment] do
+    SitemapGenerator::Sitemap.verbose = verbose
+    SitemapGenerator::Sitemap.create
+  end
+end

data/templates/sitemap.rb

@@ -0,0 +1,42 @@
+# Set the host name for URL creation
+SitemapGenerator::Sitemap.default_host = "http://www.example.com"
+
+SitemapGenerator::Sitemap.add_links do |sitemap|
+  # Put links creation logic here.
+  #
+  # The root path '/' and sitemap index file are added automatically.
+  # Links are added to the Sitemap in the order they are specified.
+  #
+  # Usage: sitemap.add path, options
+  #        (default options are used if you don't specify)
+  #
+  # Defaults: :priority => 0.5, :changefreq => 'weekly',
+  #           :lastmod => Time.now, :host => default_host
+
+
+  # Examples:
+
+  # add '/articles'
+  sitemap.add articles_path, :priority => 0.7, :changefreq => 'daily'
+
+  # add all individual articles
+  Article.find(:all).each do |a|
+    sitemap.add article_path(a), :lastmod => a.updated_at
+  end
+
+  # add merchant path
+  sitemap.add '/purchase', :priority => 0.7, :host => "https://www.example.com"
+
+end
+
+# Including Sitemaps from Rails Engines.
+#
+# These Sitemaps should be almost identical to a regular Sitemap file except
+# they needn't define their own SitemapGenerator::Sitemap.default_host since
+# they will undoubtedly share the host name of the application they belong to.
+#
+# As an example, say we have a Rails Engine in vendor/plugins/cadability_client
+# We can include its Sitemap here as follows:
+#
+# file = File.join(Rails.root, 'vendor/plugins/cadability_client/config/sitemap.rb')
+# eval(open(file).read, binding, file)

metadata

@@ -0,0 +1,133 @@
+--- !ruby/object:Gem::Specification 
+name: apsoto-sitemap_generator
+version: !ruby/object:Gem::Version 
+  hash: 858800296
+  prerelease: true
+  segments: 
+  - 1
+  - 0
+  - 1
+  - dev
+  version: 1.0.1.dev
+platform: ruby
+authors: 
+- Alex Soto
+- Karl Varga
+- Adam Salter
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2010-09-13 00:00:00 -07:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: rspec
+  prerelease: false
+  requirement: &id001 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 3
+        segments: 
+        - 0
+        version: "0"
+  type: :development
+  version_requirements: *id001
+- !ruby/object:Gem::Dependency 
+  name: nokogiri
+  prerelease: false
+  requirement: &id002 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 3
+        segments: 
+        - 0
+        version: "0"
+  type: :development
+  version_requirements: *id002
+- !ruby/object:Gem::Dependency 
+  name: sqlite3-ruby
+  prerelease: false
+  requirement: &id003 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 3
+        segments: 
+        - 0
+        version: "0"
+  type: :development
+  version_requirements: *id003
+description: SitemapGenerator is a Rails gem that makes it easy to generate enterprise-class Sitemaps readable by all search engines.  Generated Sitemaps adhere to the Sitemap protocol specification.  When you generate new Sitemaps, SitemapGenerator can automatically ping the major search engines (including Google, Yahoo and Bing) to notify them.  SitemapGenerator includes rake tasks to easily manage your sitemaps.
+email: apsoto@gmail.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: 
+- README.md
+files: 
+- MIT-LICENSE
+- README.md
+- Rakefile
+- VERSION
+- lib/sitemap_generator.rb
+- lib/sitemap_generator/builder.rb
+- lib/sitemap_generator/builder/helper.rb
+- lib/sitemap_generator/builder/sitemap_file.rb
+- lib/sitemap_generator/builder/sitemap_index_file.rb
+- lib/sitemap_generator/interpreter.rb
+- lib/sitemap_generator/link.rb
+- lib/sitemap_generator/link_set.rb
+- lib/sitemap_generator/mapper.rb
+- lib/sitemap_generator/railtie.rb
+- lib/sitemap_generator/tasks.rb
+- lib/sitemap_generator/templates.rb
+- lib/sitemap_generator/utilities.rb
+- rails/install.rb
+- rails/uninstall.rb
+- tasks/sitemap_generator_tasks.rake
+- templates/sitemap.rb
+has_rdoc: true
+homepage: http://github.com/apsoto/sitemap_generator
+licenses: []
+
+post_install_message: 
+rdoc_options: 
+- --charset=UTF-8
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">"
+    - !ruby/object:Gem::Version 
+      hash: 25
+      segments: 
+      - 1
+      - 3
+      - 1
+      version: 1.3.1
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.3.7
+signing_key: 
+specification_version: 3
+summary: Easily generate enterprise class Sitemaps for your Rails site using a familiar Rails Routes-like DSL
+test_files: []
+