alexrabarts-big_sitemap 0.2.1 → 0.3.0

data/History.txt

@@ -1,3 +1,14 @@
+=== 0.3.0 / 2009-04-06
+
+* API change: Pass model through as first argument to add method, e.g.sitemap.add(Posts, {:path => 'articles'})
+* API change: Use Rails' polymorphic_url helper to generate URLs if Rails is being used
+* API change: Only ping search engines when ping_search_engines is explicitly called
+* Add support for passing options through to the model's find method, e.g. :conditions
+* Allow base URL to be specified as a hash as well as a string
+* Add support for changefreq and priority
+* Pluralize sitemap model filenames
+* GZipping may optionally be turned off
+
 === 0.2.1 / 2009-03-12
 
 * Normalize path arguments so it no longer matters whether a leading slash is used or not

data/LICENSE

@@ -0,0 +1,22 @@
+(The MIT License)
+
+Copyright (c) 2009 Stateless Systems (http://statelesssystems.com)
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -1,104 +1,110 @@
 = BigSitemap
 
-== DESCRIPTION
+BigSitemap is a Sitemap (http://sitemaps.org) generator suitable for applications with greater than 50,000 URLs.  It splits large Sitemaps into multiple files, gzips the files to minimize bandwidth usage, batches database queries to minimize memory usage, can be set up with just a few lines of code and is compatible with just about any framework.
 
-BigSitemap is a Sitemap (http://sitemaps.org) generator specifically designed for large sites (although it works equally well with small sites).  It splits large Sitemaps into multiple files, gzips the files to minimize bandwidth usage, batches database queries so it doesn't take your site down, can be set up with just a few lines of code and is compatible with just about any framework.
+BigSitemap is best run periodically through a Rake/Thor task.
 
-== INSTALL
+  sitemap = BigSitemap.new(:url_options => {:host => 'example.com'})
 
-Via git:
+  # Add a model
+  sitemap.add Product
 
-  git clone git://github.com/alexrabarts/big_sitemap.git
+  # Add another model with some options
+  sitemap.add(Post, {
+    :conditions       => {:published => true},
+    :path             => 'articles',
+    :change_frequency => 'daily',
+    :priority         => 0.5
+  })
 
-Via gem:
-
-  gem install alexrabarts-big_sitemap -s http://gems.github.com
-
-== SYNOPSIS
+  # Generate the files
+  sitemap.generate
 
-The minimum required to generate a sitemap is:
+The code above will create a minimum of three files:
 
-  BigSitemap.new(:base_url => 'http://example.com').add(:model => MyModel, :path => 'my_controller').generate
+1. public/sitemaps/sitemap_index.xml.gz
+2. public/sitemaps/sitemap_products.xml.gz
+3. public/sitemaps/sitemap_posts.xml.gz
 
-You can put this in a rake/thor task and create a cron job to run it periodically.  It should be enough for most Rails/Merb applications.  You can add more models by further calls to the <code>add</code> method.  Note that the methods are chainable, although you can call them on an instance variable if you prefer:
+If your sitemaps grow beyond 50,000 URLs (this limit can be overridden with the <code>:max_per_sitemap</code> option), the sitemap files will be partitioned into multiple files (<code>sitemap_products_1.xml.gz</code>, <code>sitemap_products_2.xml.gz</code>, &#133;).
 
-  sitemap = BigSitemap.new(:base_url => 'http://example.com')
-  sitemap.add(:model => Posts,    :path => 'articles')
-  sitemap.add(:model => Comments, :path => 'comments')
-  sitemap.generate
+If you're using Rails then the URLs for each database record are generated with the <code>polymorphic_url</code> helper.  That means that the URL for a record will be exactly what you would expect: generated with respect to the routing setup of your app.  In other contexts where this helper isn't available, the URLs are generated in the form:
 
-=== Find Methods
+  :base_url/:path/:to_param
 
-Your models must provide either a <code>find_for_sitemap</code> or <code>all</code> class method that returns the instances that are to be included in the sitemap.
+If the <code>to_param</code> method does not exist, then <code>id</code> will be used.
 
-Additionally, you models must provide a <code>count_for_sitemap</code> or <code>count</code> class method that returns a count of the instances to be included.
+== Install
 
-If you're using ActiveRecord (Rails) or DataMapper then <code>all</code> and <code>count</code> are already provided and you don't need to do anything unless you want to include a subset of records.  If you provide your own <code>find_for_sitemap</code> or <code>all</code> method then it should be able to handle the <code>:offset</code> and <code>:limit</code> options, in the same way that ActiveRecord and DataMapper handle them.  This is especially important if you have more than 50,000 URLs.
+Via gem:
 
-=== URL Format
+  gem install alexrabarts-big_sitemap -s http://gems.github.com
 
-To generate the URLs, BigSitemap will combine the constructor arguments with the <code>to_param</code> method of each instance returned (provided by ActiveRecord but not DataMapper).  If this method is not present, <code>id</code> will be used.  The URL is constructed as:
+== Advanced
 
-  :base_url/:path/:to_param (if to_param exists)
-  :base_url/:path/:id       (if to_param does not exist)
+=== Options
 
-=== Sitemap Location
+  * <code>:url_options</code> -- hash with <code>:host</code>, optionally <code>:port</code> and <code>:protocol</code>
+  * <code>:base_url</code> -- string alternative to <code>:url_options</code>, e.g. "https://example.com:8080/"
+  * <code>:document_root</code> -- string defaults to <code>Rails.root</code> or <code>Merb.root</code> if available
+  * <code>:path</code> -- string defaults to 'sitemaps', which places sitemap files under the <code>/sitemaps</code> directory
+  * <code>:max_per_sitemap</code> -- <code>50000</code>, which is the limit dictated by Google but can be less
+  * <code>:batch_size</code> -- <code>1001</code> (not <code>1000</code> due to a bug in DataMapper)
+  * <code>:gzip</code> -- <code>true</code>
+  * <code>:ping_google</code> -- <code>true</code>
+  * <code>:ping_yahoo</code> -- <code>false</code>, needs <code>:yahoo_app_id</code>
+  * <code>:ping_msn</code> -- <code>false</code>
+  * <code>:pink_ask</code> -- <code>false</code>
 
-BigSitemap knows about the document root of Rails and Merb.  If you are using another framework then you can specify the document root with the <code>:document_root</code> option.  e.g.:
+=== Chaining
 
-  BigSitemap.new(:base_url => 'http://example.com', :document_root => "#{FOO_ROOT}/httpdocs")
+You can chain methods together.  You could even get away with as little code as:
 
-By default, the sitemap files are created under <code>/sitemaps</code>.  You can modify this with the <code>:path</code> option:
+  BigSitemap.new(:url_options => {:host => 'example.com'}).add(Post).generate
 
-  BigSitemap.new(:base_url => 'http://example.com', :path => 'google-sitemaps') # places Sitemaps under /google-sitemaps
+=== Pinging Search Engines
 
-=== Cleaning the Sitemaps Directory
+To ping search engines, call <code>ping_search_engines</code> after you generate the sitemap:
 
-Calling the <code>clean</code> method will remove all files from the Sitemaps directory.
-
-=== Maximum Number of URLs
-
-Sitemaps will be split across several files if more than 50,000 records are returned.  You can customize this limit with the <code>:max_per_sitemap</code> option:
+  sitemap.generate
+  sitemap.ping_search_engines
 
-  BigSitemap.new(:base_url => 'http://example.com', :max_per_sitemap => 1000) # Max of 1000 URLs per Sitemap
+=== Change Frequency and Priority
 
-=== Batched Database Queries
+You can control "changefreq" and "priority" values for each record individually by passing lambdas instead of fixed values:
 
-The database is queried in batches to prevent large SQL select statements from locking the database for too long.  By default, the batch size is 1001 (not 1000 due to an obscure bug in DataMapper).  You can customize the batch size with the <code>:batch_size</code> option:
+  sitemap.add(Posts,
+    :change_frequency => lambda {|post| ... },
+    :priority         => lambda {|post| ... }
+  )
 
-  BigSitemap.new(:base_url => 'http://example.com', :batch_size => 5000) # Database is queried in batches of 5,000
+=== Find Methods
 
-=== Search Engine Notification
+Your models must provide either a <code>find_for_sitemap</code> or <code>all</code> class method that returns the instances that are to be included in the sitemap.
 
-Google, Yahoo!, MSN and Ask are pinged once the Sitemap files are generated.  You can turn one or more of these off:
+Additionally, you models must provide a <code>count_for_sitemap</code> or <code>count</code> class method that returns a count of the instances to be included.
 
-  BigSitemap.new(
-    :base_url => 'http://example.com',
-    :ping_google => false,
-    :ping_yahoo  => false,
-    :ping_msn    => false,
-    :ping_ask    => false
-  )
+If you're using ActiveRecord (Rails) or DataMapper then <code>all</code> and <code>count</code> are already provided and you don't need to do anything unless you want to include a subset of records.  If you provide your own <code>find_for_sitemap</code> or <code>all</code> method then it should be able to handle the <code>:offset</code> and <code>:limit</code> options, in the same way that ActiveRecord and DataMapper handle them.  This is especially important if you have more than 50,000 URLs.
 
-You must provide an App ID in order to ping Yahoo! (more info at http://developer.yahoo.com/search/siteexplorer/V1/updateNotification.html):
+=== Cleaning the Sitemaps Directory
 
-  BigSitemap.new(:base_url => 'http://example.com', :yahoo_app_id => 'myYahooAppId') # Yahoo! will now be pinged
+Calling the <code>clean</code> method will remove all files from the Sitemaps directory.
 
-== LIMITATIONS
+== Limitations
 
 If your database is likely to shrink during the time it takes to create the sitemap then you might run into problems (the final, batched SQL select will overrun by setting a limit that is too large since it is calculated from the count, which is queried at the very beginning).  Patches welcome!
 
 == TODO
 
-* Support for <code>priority</code>
-* Support for <code>changefreq</code> (currently hard-coded to <code>weekly</code>)
+Tests for Rails components.
 
-== CREDITS
+== Credits
 
 Thanks to Alastair Brunton and Harry Love, who's work provided a starting point for this library.
 http://scoop.cheerfactory.co.uk/2008/02/26/google-sitemap-generator/
 
-== COPYRIGHT
+Thanks to Mislav Marohnić for contributing patches.
 
-Copyright (c) 2009 Stateless Systems (http://statelesssystems.com). See LICENSE for details.
+== Copyright
 
+Copyright (c) 2009 Stateless Systems (http://statelesssystems.com). See LICENSE for details.

data/VERSION.yml

@@ -1,4 +1,4 @@
 --- 
-:minor: 2
-:patch: 1
+:patch: 0
 :major: 0
+:minor: 3

data/lib/big_sitemap.rb

@@ -1,82 +1,100 @@
-require 'net/http'
 require 'uri'
 require 'zlib'
 require 'builder'
 require 'extlib'
 
 class BigSitemap
+  DEFAULTS = {
+    :max_per_sitemap => 50000,
+    :batch_size      => 1001,
+    :path            => 'sitemaps',
+    :gzip            => true,
+
+    # opinionated
+    :ping_google => true,
+    :ping_yahoo  => false, # needs :yahoo_app_id
+    :ping_msn    => false,
+    :ping_ask    => false
+  }
+
+  COUNT_METHODS     = [:count_for_sitemap, :count]
+  FIND_METHODS      = [:find_for_sitemap, :all]
+  TIMESTAMP_METHODS = [:updated_at, :updated_on, :updated, :created_at, :created_on, :created]
+  PARAM_METHODS     = [:to_param, :id]
+
+  include ActionController::UrlWriter if defined? Rails
+
   def initialize(options)
-    document_root = options.delete(:document_root)
+    @options = DEFAULTS.merge options
+
+    # Use Rails' default_url_options if available
+    @default_url_options = defined?(Rails) ? default_url_options : {}
+
+    if @options[:url_options]
+      @default_url_options.update @options[:url_options]
+    elsif @options[:base_url]
+      uri = URI.parse(@options[:base_url])
+      @default_url_options[:host]     = uri.host
+      @default_url_options[:port]     = uri.port
+      @default_url_options[:protocol] = uri.scheme
+    else
+      raise ArgumentError, 'you must specify either ":url_options" hash or ":base_url" string'
+    end
+
+    if @options[:batch_size] > @options[:max_per_sitemap]
+      raise ArgumentError, '":batch_size" must be less than ":max_per_sitemap"'
+    end
 
-    if document_root.nil?
-      if defined? RAILS_ROOT
-        document_root = "#{RAILS_ROOT}/public"
+    @options[:document_root] ||= begin
+      if defined? Rails
+        "#{Rails.root}/public"
       elsif defined? Merb
-        document_root = "#{Merb.root}/public"
+        "#{Merb.root}/public"
       end
     end
 
-    raise ArgumentError, 'Document root must be specified with the :document_root option' if document_root.nil?
-
-    @base_url        = options.delete(:base_url)
-    @max_per_sitemap = options.delete(:max_per_sitemap) || 50000
-    @batch_size      = options.delete(:batch_size) || 1001 # TODO: Set this to 1000 once DM offset 37000 bug is fixed
-    @web_path        = strip_leading_slash(options.delete(:path) || 'sitemaps')
-    @ping_google     = options[:ping_google].nil? ? true : options.delete(:ping_google)
-    @ping_yahoo      = options[:ping_yahoo].nil? ? true : options.delete(:ping_yahoo)
-    @yahoo_app_id    = options.delete(:yahoo_app_id)
-    @ping_msn        = options[:ping_msn].nil? ? true : options.delete(:ping_msn)
-    @ping_ask        = options[:ping_ask].nil? ? true : options.delete(:ping_ask)
-    @file_path       = "#{document_root}/#{@web_path}"
-    @sources         = []
-
-    raise ArgumentError, "Base URL must be specified with the :base_url option" if @base_url.nil?
-
-    raise(
-      ArgumentError,
-      'Batch size (:batch_size) must be less than or equal to maximum URLs per sitemap (:max_per_sitemap)'
-    ) if @batch_size > @max_per_sitemap
+    unless @options[:document_root]
+      raise ArgumentError, 'Document root must be specified with the ":document_root" option'
+    end
 
+    @file_path = "#{@options[:document_root]}/#{strip_leading_slash(@options[:path])}"
     Dir.mkdir(@file_path) unless File.exists? @file_path
+
+    @sources       = []
+    @sitemap_files = []
   end
 
-  def add(options)
-    raise ArgumentError, ':model and :path options must be provided' unless options[:model] && options[:path]
-    @sources << options.update(:path => strip_leading_slash(options[:path]))
-    self # Chainable
+  def add(model, options={})
+    options[:path] ||= Extlib::Inflection.tableize(model.to_s)
+    @sources << [model, options.dup]
+    return self
   end
 
   def clean
-    unless @file_path.nil?
-      Dir.foreach(@file_path) do |f|
-        f = "#{@file_path}/#{f}"
-        File.delete(f) if File.file?(f)
-      end
+    Dir["#{@file_path}/sitemap_*.{xml,xml.gz}"].each do |file|
+      FileUtils.rm file
     end
-    self # Chainable
+    return self
   end
 
   def generate
-    @sources.each do |source|
-      klass = source[:model]
+    for model, options in @sources
+      count_method = pick_method(model, COUNT_METHODS)
+      find_method  = pick_method(model, FIND_METHODS)
+      raise ArgumentError, "#{model} must provide a count_for_sitemap class method" if count_method.nil?
+      raise ArgumentError, "#{model} must provide a find_for_sitemap class method" if find_method.nil?
 
-      count_method = pick_method(klass, [:count_for_sitemap, :count])
-      find_method  = pick_method(klass, [:find_for_sitemap, :all])
-      raise ArgumentError, "#{klass} must provide a count_for_sitemap class method" if count_method.nil?
-      raise ArgumentError, "#{klass} must provide a find_for_sitemap class method" if find_method.nil?
-
-      count        = klass.send(count_method)
+      count        = model.send(count_method)
       num_sitemaps = 1
       num_batches  = 1
 
-      if count > @batch_size
-        num_batches  = (count.to_f / @batch_size.to_f).ceil
-        num_sitemaps = (count.to_f / @max_per_sitemap.to_f).ceil
+      if count > @options[:batch_size]
+        num_batches  = (count.to_f / @options[:batch_size].to_f).ceil
+        num_sitemaps = (count.to_f / @options[:max_per_sitemap].to_f).ceil
       end
       batches_per_sitemap = num_batches.to_f / num_sitemaps.to_f
 
-      # Update the @sources hash so that the index file knows how many sitemaps to link to
-      source[:num_sitemaps] = num_sitemaps
+      find_options = options.dup
 
       for sitemap_num in 1..num_sitemaps
         # Work out the start and end batch numbers for this sitemap
@@ -84,126 +102,147 @@ class BigSitemap
         batch_num_end   = (batch_num_start + [batches_per_sitemap, num_batches].min).floor - 1
 
         # Stream XML output to a file
-        filename = "sitemap_#{Extlib::Inflection::underscore(klass.to_s)}"
+        filename = "sitemap_#{Extlib::Inflection::tableize(model.to_s)}"
         filename << "_#{sitemap_num}" if num_sitemaps > 1
 
-        gz = gz_writer("#{filename}.xml.gz")
+        f = xml_open(filename)
 
-        xml = Builder::XmlMarkup.new(:target => gz)
+        xml = Builder::XmlMarkup.new(:target => f)
         xml.instruct!
         xml.urlset(:xmlns => 'http://www.sitemaps.org/schemas/sitemap/0.9') do
           for batch_num in batch_num_start..batch_num_end
-            offset       = ((batch_num - 1) * @batch_size)
-            limit        = (count - offset) < @batch_size ? (count - offset - 1) : @batch_size
-            find_options = num_batches > 1 ? {:limit => limit, :offset => offset} : {}
-
-            klass.send(find_method, find_options).each do |r|
-              last_mod_method = pick_method(
-                r,
-                [:updated_at, :updated_on, :updated, :created_at, :created_on, :created]
-              )
+            offset       = ((batch_num - 1) * @options[:batch_size])
+            limit        = (count - offset) < @options[:batch_size] ? (count - offset - 1) : @options[:batch_size]
+            find_options.update(:limit => limit, :offset => offset) if num_batches > 1
+
+            model.send(find_method, find_options).each do |r|
+              last_mod_method = pick_method(r, TIMESTAMP_METHODS)
               last_mod = last_mod_method.nil? ? Time.now : r.send(last_mod_method)
 
-              param_method = pick_method(r, [:to_param, :id])
-              raise ArgumentError, "#{klass} must provide a to_param instance method" if param_method.nil?
+              param_method = pick_method(r, PARAM_METHODS)
 
               xml.url do
-                xml.loc("#{@base_url}/#{source[:path]}/#{r.send(param_method)}")
+                location = defined?(Rails) ?
+                  polymorphic_url(r) :
+                  "#{root_url}/#{strip_leading_slash(options[:path])}/#{r.send(param_method)}"
+                xml.loc(location)
+
                 xml.lastmod(last_mod.strftime('%Y-%m-%d')) unless last_mod.nil?
-                xml.changefreq('weekly')
+
+                change_frequency = options[:change_frequency] || 'weekly'
+                xml.changefreq(change_frequency.is_a?(Proc) ? change_frequency.call(r) : change_frequency)
+
+                priority = options[:priority]
+                unless priority.nil?
+                  xml.priority(priority.is_a?(Proc) ? priority.call(r) : priority)
+                end
               end
             end
           end
         end
 
-        gz.close
+        f.close
       end
 
     end
 
     generate_sitemap_index
-    ping_search_engines
-    self # Chainable
+
+    return self
   end
 
-  private
-    def strip_leading_slash(str)
-      str.sub(/^\//, '')
+  def ping_search_engines
+    require 'net/http'
+    require 'cgi'
+
+    sitemap_uri = CGI::escape(url_for_sitemap(@sitemap_files.last))
+
+    if @options[:ping_google]
+      Net::HTTP.get('www.google.com', "/webmasters/tools/ping?sitemap=#{sitemap_uri}")
     end
 
-    def pick_method(klass, candidates)
-      method = nil
-      candidates.each do |candidate|
-        if klass.respond_to? candidate
-          method = candidate
-          break
-        end
+    if @options[:ping_yahoo]
+      if @options[:yahoo_app_id]
+        Net::HTTP.get(
+          'search.yahooapis.com', "/SiteExplorerService/V1/updateNotification?" +
+            "appid=#{@options[:yahoo_app_id]}&url=#{sitemap_uri}"
+        )
+      else
+        $stderr.puts 'unable to ping Yahoo: no ":yahoo_app_id" provided'
       end
-      method
     end
 
-    def gz_writer(filename)
-      Zlib::GzipWriter.new(File.open("#{@file_path}/#{filename}", 'w+'))
+    if @options[:ping_msn]
+      Net::HTTP.get('webmaster.live.com', "/ping.aspx?siteMap=#{sitemap_uri}")
     end
 
-    def sitemap_index_filename
-      'sitemap_index.xml.gz'
+    if @options[:pink_ask]
+      Net::HTTP.get('submissions.ask.com', "/ping?sitemap=#{sitemap_uri}")
     end
+  end
 
-    # Create a sitemap index document
-    def generate_sitemap_index
-      xml     = ''
-      builder = Builder::XmlMarkup.new(:target => xml)
-      builder.instruct!
-      builder.sitemapindex(:xmlns => 'http://www.sitemaps.org/schemas/sitemap/0.9') do
-        @sources.each do |source|
-          num_sitemaps = source[:num_sitemaps]
-          for i in 1..num_sitemaps
-            loc = "#{@base_url}/#{@web_path}/sitemap_#{Extlib::Inflection::underscore(source[:model].to_s)}"
-            loc << "_#{i}" if num_sitemaps > 1
-            loc << '.xml.gz'
-
-            builder.sitemap do
-              builder.loc(loc)
-              builder.lastmod(Time.now.strftime('%Y-%m-%d'))
-            end
-          end
-        end
-      end
-
-      gz = gz_writer(sitemap_index_filename)
-      gz.write(xml)
-      gz.close
+  def root_url
+    @root_url ||= begin
+      url = ''
+      url << (@default_url_options[:protocol] || 'http')
+      url << '://' unless url.match('://')
+      url << @default_url_options[:host]
+      url << ":#{port}" if port = @default_url_options[:port] and port != 80
     end
+  end
 
-    def sitemap_uri
-      URI.escape("#{@base_url}/#{@web_path}/#{sitemap_index_filename}")
-    end
+  private
 
-    # Notify Google of the new sitemap index file
-    def ping_google
-      Net::HTTP.get('www.google.com', "/webmasters/tools/ping?sitemap=#{sitemap_uri}")
-    end
+  def strip_leading_slash(str)
+    str.sub(/^\//, '')
+  end
 
-    # Notify Yahoo! of the new sitemap index file
-    def ping_yahoo
-      Net::HTTP.get('search.yahooapis.com', "/SiteExplorerService/V1/updateNotification?appid=#{@yahoo_app_id}&url=#{sitemap_uri}")
+  def pick_method(model, candidates)
+    method = nil
+    candidates.each do |candidate|
+      if model.respond_to? candidate
+        method = candidate
+        break
+      end
     end
+    method
+  end
 
-    # Notify MSN of the new sitemap index file
-    def ping_msn
-      Net::HTTP.get('webmaster.live.com', "/ping.aspx?siteMap=#{sitemap_uri}")
-    end
+  def xml_open(filename)
+    filename << '.xml'
+    filename << '.gz' if @options[:gzip]
 
-    # Notify Ask of the new sitemap index file
-    def ping_ask
-      Net::HTTP.get('submissions.ask.com', "/ping?sitemap=#{sitemap_uri}")
+    file = File.open("#{@file_path}/#{filename}", 'w+')
+
+    @sitemap_files << file.path
+
+    writer = @options[:gzip] ? Zlib::GzipWriter.new(file) : file
+
+    if block_given?
+      yield writer
+      writer.close
     end
 
-    def ping_search_engines
-      ping_google if @ping_google
-      ping_yahoo if @ping_yahoo && @yahoo_app_id
-      ping_msn if @ping_msn
-      ping_ask if @ping_ask
+    writer
+  end
+
+  def url_for_sitemap(path)
+    "#{root_url}/#{File.basename(path)}"
+  end
+
+  # Create a sitemap index document
+  def generate_sitemap_index
+    xml_open 'sitemap_index' do |file|
+      xml = Builder::XmlMarkup.new(:target => file)
+      xml.instruct!
+      xml.sitemapindex(:xmlns => 'http://www.sitemaps.org/schemas/sitemap/0.9') do
+        for path in @sitemap_files[0..-2]
+          xml.sitemap do
+            xml.loc(url_for_sitemap(path))
+            xml.lastmod(Time.now.strftime('%Y-%m-%d'))
+          end
+        end
+      end
     end
-end
+  end
+end

data/test/big_sitemap_test.rb

@@ -15,6 +15,14 @@ class BigSitemapTest < Test::Unit::TestCase
     assert_raise(ArgumentError) { BigSitemap.new(:document_root => tmp_dir) }
   end
 
+  should 'generate the same base URL' do
+    options = {:document_root => tmp_dir}
+    assert_equal(
+      BigSitemap.new(options.merge(:base_url => 'http://example.com')).root_url,
+      BigSitemap.new(options.merge(:url_options => {:host => 'example.com'})).root_url
+    )
+  end
+
   should 'generate a sitemap index file' do
     generate_sitemap_files
     assert File.exists?(sitemaps_index_file)
@@ -27,21 +35,14 @@ class BigSitemapTest < Test::Unit::TestCase
     assert File.exists?(single_sitemaps_model_file), "#{single_sitemaps_model_file} exists"
   end
 
-  should 'generate exactly two sitemap model files' do
-    generate_exactly_two_model_sitemap_files
+  should 'generate two sitemap model files' do
+    generate_two_model_sitemap_files
     assert File.exists?(first_sitemaps_model_file), "#{first_sitemaps_model_file} exists"
     assert File.exists?(second_sitemaps_model_file), "#{second_sitemaps_model_file} exists"
     third_sitemaps_model_file = "#{sitemaps_dir}/sitemap_test_model_3.xml.gz"
     assert !File.exists?(third_sitemaps_model_file), "#{third_sitemaps_model_file} does not exist"
   end
 
-  should 'clean all sitemap files' do
-    generate_sitemap_files
-    assert Dir.entries(sitemaps_dir).size > 2, "#{sitemaps_dir} is not empty" # ['.', '..'].size == 2
-    @sitemap.clean
-    assert_equal 2, Dir.entries(sitemaps_dir).size, "#{sitemaps_dir} is empty"
-  end
-
   context 'Sitemap index file' do
     should 'contain one sitemapindex element' do
       generate_sitemap_files
@@ -54,79 +55,125 @@ class BigSitemapTest < Test::Unit::TestCase
     end
 
     should 'contain one loc element' do
-      generate_sitemap_files
+      generate_one_sitemap_model_file
       assert_equal 1, num_elements(sitemaps_index_file, 'loc')
     end
 
     should 'contain one lastmod element' do
-      generate_sitemap_files
+      generate_one_sitemap_model_file
       assert_equal 1, num_elements(sitemaps_index_file, 'lastmod')
     end
 
     should 'contain two loc elements' do
-      generate_exactly_two_model_sitemap_files
+      generate_two_model_sitemap_files
       assert_equal 2, num_elements(sitemaps_index_file, 'loc')
     end
 
     should 'contain two lastmod elements' do
-      generate_exactly_two_model_sitemap_files
+      generate_two_model_sitemap_files
       assert_equal 2, num_elements(sitemaps_index_file, 'lastmod')
     end
+
+    should 'not be gzipped' do
+      generate_sitemap_files(:gzip => false)
+      assert File.exists?(unzipped_sitemaps_index_file)
+    end
   end
 
   context 'Sitemap model file' do
     should 'contain one urlset element' do
-      generate_sitemap_files
+      generate_one_sitemap_model_file
       assert_equal 1, num_elements(single_sitemaps_model_file, 'urlset')
     end
 
     should 'contain several loc elements' do
-      generate_sitemap_files
+      generate_one_sitemap_model_file
       assert_equal default_num_items, num_elements(single_sitemaps_model_file, 'loc')
     end
 
     should 'contain several lastmod elements' do
-      generate_sitemap_files
+      generate_one_sitemap_model_file
       assert_equal default_num_items, num_elements(single_sitemaps_model_file, 'lastmod')
     end
 
     should 'contain several changefreq elements' do
-      generate_sitemap_files
+      generate_one_sitemap_model_file
       assert_equal default_num_items, num_elements(single_sitemaps_model_file, 'changefreq')
     end
 
+    should 'contain several priority elements' do
+      generate_one_sitemap_model_file(:priority => 0.2)
+      assert_equal default_num_items, num_elements(single_sitemaps_model_file, 'priority')
+    end
+
+    should 'have a change frequency of weekly by default' do
+      generate_one_sitemap_model_file
+      assert_equal 'weekly', elements(single_sitemaps_model_file, 'changefreq').first.text
+    end
+
+    should 'have a change frequency of daily' do
+      generate_one_sitemap_model_file(:change_frequency => 'daily')
+      assert_equal 'daily', elements(single_sitemaps_model_file, 'changefreq').first.text
+    end
+
+    should 'be able to use a lambda to specify change frequency' do
+      generate_one_sitemap_model_file(:change_frequency => lambda {|m| m.change_frequency})
+      assert_equal TestModel.new.change_frequency, elements(single_sitemaps_model_file, 'changefreq').first.text
+    end
+
+    should 'have a priority of 0.2' do
+      generate_one_sitemap_model_file(:priority => 0.2)
+      assert_equal '0.2', elements(single_sitemaps_model_file, 'priority').first.text
+    end
+
+    should 'be able to use a lambda to specify priority' do
+      generate_one_sitemap_model_file(:priority => lambda {|m| m.priority})
+      assert_equal TestModel.new.priority.to_s, elements(single_sitemaps_model_file, 'priority').first.text
+    end
+
     should 'contain one loc element' do
-      generate_exactly_two_model_sitemap_files
+      generate_two_model_sitemap_files
       assert_equal 1, num_elements(first_sitemaps_model_file, 'loc')
       assert_equal 1, num_elements(second_sitemaps_model_file, 'loc')
     end
 
     should 'contain one lastmod element' do
-      generate_exactly_two_model_sitemap_files
+      generate_two_model_sitemap_files
       assert_equal 1, num_elements(first_sitemaps_model_file, 'lastmod')
       assert_equal 1, num_elements(second_sitemaps_model_file, 'lastmod')
     end
 
     should 'contain one changefreq element' do
-      generate_exactly_two_model_sitemap_files
+      generate_two_model_sitemap_files
       assert_equal 1, num_elements(first_sitemaps_model_file, 'changefreq')
       assert_equal 1, num_elements(second_sitemaps_model_file, 'changefreq')
     end
 
+    should 'contain one priority element' do
+      generate_two_model_sitemap_files(:priority => 0.2)
+      assert_equal 1, num_elements(first_sitemaps_model_file, 'priority')
+      assert_equal 1, num_elements(second_sitemaps_model_file, 'priority')
+    end
+
     should 'strip leading slashes from controller paths' do
       create_sitemap
-      @sitemap.add(:model => TestModel, :path => '/test_controller').generate
+      @sitemap.add(TestModel, :path => '/test_controller').generate
       assert(
         !elements(single_sitemaps_model_file, 'loc').first.text.match(/\/\/test_controller\//),
         'URL does not contain a double-slash before the controller path'
       )
     end
+
+    should 'not be gzipped' do
+      generate_one_sitemap_model_file(:gzip => false)
+      assert File.exists?(unzipped_single_sitemaps_model_file)
+    end
   end
 
   context 'add method' do
     should 'be chainable' do
       create_sitemap
-      assert_equal BigSitemap, @sitemap.add(:model => TestModel, :path => 'test_controller').class
+      assert_equal BigSitemap, @sitemap.add(TestModel).class
     end
   end
 
@@ -135,6 +182,13 @@ class BigSitemapTest < Test::Unit::TestCase
       create_sitemap
       assert_equal BigSitemap, @sitemap.clean.class
     end
+
+    should 'clean all sitemap files' do
+      generate_sitemap_files
+      assert Dir.entries(sitemaps_dir).size > 2, "#{sitemaps_dir} is not empty" # ['.', '..'].size == 2
+      @sitemap.clean
+      assert_equal 2, Dir.entries(sitemaps_dir).size, "#{sitemaps_dir} is empty"
+    end
   end
 
   context 'generate method' do
@@ -157,22 +211,32 @@ class BigSitemapTest < Test::Unit::TestCase
       }.update(options))
     end
 
-    def generate_sitemap_files
-      create_sitemap
+    def generate_sitemap_files(options={})
+      create_sitemap(options)
       add_model
       @sitemap.generate
     end
 
-    def generate_exactly_two_model_sitemap_files
-      create_sitemap(:max_per_sitemap => 1, :batch_size => 1)
-      add_model(:num_items => 2)
+    def generate_one_sitemap_model_file(options={})
+      change_frequency = options.delete(:change_frequency)
+      priority         = options.delete(:priority)
+      create_sitemap(options.merge(:max_per_sitemap => default_num_items, :batch_size => default_num_items))
+      add_model(:change_frequency => change_frequency, :priority => priority)
+      @sitemap.generate
+    end
+
+    def generate_two_model_sitemap_files(options={})
+      change_frequency = options.delete(:change_frequency)
+      priority         = options.delete(:priority)
+      create_sitemap(options.merge(:max_per_sitemap => 1, :batch_size => 1))
+      add_model(:num_items => 2, :change_frequency => change_frequency, :priority => priority)
       @sitemap.generate
     end
 
     def add_model(options={})
       num_items = options.delete(:num_items) || default_num_items
       TestModel.stubs(:num_items).returns(num_items)
-      @sitemap.add({:model => TestModel, :path => 'test_controller'}.update(options))
+      @sitemap.add(TestModel, options)
     end
 
     def default_num_items
@@ -180,19 +244,27 @@ class BigSitemapTest < Test::Unit::TestCase
     end
 
     def sitemaps_index_file
-      "#{sitemaps_dir}/sitemap_index.xml.gz"
+      "#{unzipped_sitemaps_index_file}.gz"
+    end
+
+    def unzipped_sitemaps_index_file
+      "#{sitemaps_dir}/sitemap_index.xml"
     end
 
     def single_sitemaps_model_file
-      "#{sitemaps_dir}/sitemap_test_model.xml.gz"
+      "#{unzipped_single_sitemaps_model_file}.gz"
+    end
+
+    def unzipped_single_sitemaps_model_file
+      "#{sitemaps_dir}/sitemap_test_models.xml"
     end
 
     def first_sitemaps_model_file
-      "#{sitemaps_dir}/sitemap_test_model_1.xml.gz"
+      "#{sitemaps_dir}/sitemap_test_models_1.xml.gz"
     end
 
     def second_sitemaps_model_file
-      "#{sitemaps_dir}/sitemap_test_model_2.xml.gz"
+      "#{sitemaps_dir}/sitemap_test_models_2.xml.gz"
     end
 
     def sitemaps_dir
@@ -215,4 +287,4 @@ class BigSitemapTest < Test::Unit::TestCase
     def num_elements(filename, el)
       elements(filename, el).size
     end
-end
+end

data/test/fixtures/test_model.rb

@@ -3,6 +3,14 @@ class TestModel
     object_id
   end
 
+  def change_frequency
+    'monthly'
+  end
+
+  def priority
+    0.8
+  end
+
   class << self
     def count_for_sitemap
       self.find_for_sitemap.size

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: alexrabarts-big_sitemap
 version: !ruby/object:Gem::Version 
-  version: 0.2.1
+  version: 0.3.0
 platform: ruby
 authors: 
 - Alex Rabarts
@@ -9,7 +9,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2009-03-12 00:00:00 -07:00
+date: 2009-04-06 00:00:00 -07:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -38,8 +38,9 @@ executables: []
 
 extensions: []
 
-extra_rdoc_files: []
-
+extra_rdoc_files: 
+- README.rdoc
+- LICENSE
 files: 
 - History.txt
 - README.rdoc
@@ -49,6 +50,7 @@ files:
 - test/fixtures
 - test/fixtures/test_model.rb
 - test/test_helper.rb
+- LICENSE
 has_rdoc: true
 homepage: http://github.com/alexrabarts/big_sitemap
 post_install_message: