big_sitemap 0.5.1 → 0.8.1

data/.gitignore

@@ -0,0 +1,3 @@
+.rvmrc
+._*
+pkg

data/Gemfile

@@ -0,0 +1,7 @@
+source :rubygems
+
+group :test do
+  gem 'shoulda'
+  gem 'mocha'
+  gem 'nokogiri'
+end

data/Gemfile.lock

@@ -0,0 +1,16 @@
+GEM
+  remote: http://rubygems.org/
+  specs:
+    mocha (0.9.10)
+      rake
+    nokogiri (1.4.4)
+    rake (0.8.7)
+    shoulda (2.11.3)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  mocha
+  nokogiri
+  shoulda

data/README.rdoc

@@ -1,55 +1,68 @@
 = BigSitemap
 
-BigSitemap is a Sitemap (http://sitemaps.org) generator suitable for applications with greater than 50,000 URLs.  It splits large Sitemaps into multiple files, gzips the files to minimize bandwidth usage, batches database queries to minimize memory usage, can be set up with just a few lines of code and is compatible with just about any framework.
+BigSitemap is a {Sitemap}[http://sitemaps.org] generator suitable for applications with greater than 50,000 URLs.  It splits large Sitemaps into multiple files, gzips the files to minimize bandwidth usage, batches database queries to minimize memory usage, supports increment updates, can be set up with just a few lines of code and is compatible with just about any framework.
 
 BigSitemap is best run periodically through a Rake/Thor task.
 
   require 'big_sitemap'
 
-  sitemap = BigSitemap.new(:url_options => {:host => 'example.com'})
+  sitemap = BigSitemap.new(
+    :url_options   => {:host => 'example.com'},
+    :document_root => "#{APP_ROOT}/public"
+  )
 
   # Add a model
   sitemap.add Product
 
   # Add another model with some options
-  sitemap.add(Post, {
+  sitemap.add(Post,
     :conditions       => {:published => true},
     :path             => 'articles',
     :change_frequency => 'daily',
     :priority         => 0.5
-  })
+  )
+
+  # Add a static resource
+  sitemap.add_static('http://example.com/about', Time.now, 'monthly', 0.1)
 
   # Generate the files
   sitemap.generate
 
-The code above will create a minimum of three files:
+The code above will create a minimum of four files:
 
 1. public/sitemaps/sitemap_index.xml.gz
 2. public/sitemaps/sitemap_products.xml.gz
 3. public/sitemaps/sitemap_posts.xml.gz
+4. public/sitemaps/sitemap_static.xml.gz
 
 If your sitemaps grow beyond 50,000 URLs (this limit can be overridden with the <code>:max_per_sitemap</code> option), the sitemap files will be partitioned into multiple files (<code>sitemap_products_1.xml.gz</code>, <code>sitemap_products_2.xml.gz</code>, ...).
 
-If you're using Rails then the URLs for each database record are generated with the <code>polymorphic_url</code> helper.  That means that the URL for a record will be exactly what you would expect: generated with respect to the routing setup of your app.  In other contexts where this helper isn't available, the URLs are generated in the form:
+=== Framework-specific Classes
+
+Use the framework-specific classes to take advantage of built-in shortcuts.
+
+==== Rails
+
+<code>BigSiteMapRails</code> includes <code>UrlWriter</code> (useful for making use of your Rails routes - see the Location URLs section) and deals with setting the <code>:document_root</code> and <code>:url_options</code> initialization options.
 
-  :base_url/:path/:to_param
+==== Merb
 
-If the <code>to_param</code> method does not exist, then <code>id</code> will be used.
+<code>BigSitemapMerb</code> deals with setting the <code>:document_root</code> initialization option.
 
 == Install
 
 Via gem:
 
-  sudo gem install alexrabarts-big_sitemap -s http://gems.github.com
+  sudo gem install big_sitemap
 
 == Advanced
 
-=== Options
+=== Initialization Options
 
 * <code>:url_options</code> -- hash with <code>:host</code>, optionally <code>:port</code> and <code>:protocol</code>
-* <code>:base_url</code> -- string alternative to <code>:url_options</code>, e.g. "https://example.com:8080/"
-* <code>:document_root</code> -- string defaults to <code>Rails.root</code> or <code>Merb.root</code> if available
-* <code>:path</code> -- string defaults to 'sitemaps', which places sitemap files under the <code>/sitemaps</code> directory
+* <code>:base_url</code> -- string alternative to <code>:url_options</code>, e.g. <code>'https://example.com:8080/'</code>
+* <code>:document_root</code> -- string
+* <code>:path</code> -- string defaults to <code>'sitemaps'</code>, which places sitemap files under the <code>/sitemaps</code> directory
 * <code>:max_per_sitemap</code> -- <code>50000</code>, which is the limit dictated by Google but can be less
 * <code>:batch_size</code> -- <code>1001</code> (not <code>1000</code> due to a bug in DataMapper)
 * <code>:gzip</code> -- <code>true</code>
@@ -57,28 +70,44 @@ Via gem:
 * <code>:ping_yahoo</code> -- <code>false</code>, needs <code>:yahoo_app_id</code>
 * <code>:ping_bing</code> -- <code>false</code>
 * <code>:ping_ask</code> -- <code>false</code>
+* <code>:partial_update</code> -- <code>false</code>
 
 === Chaining
 
-You can chain methods together.  You could even get away with as little code as:
+You can chain methods together:
 
   BigSitemap.new(:url_options => {:host => 'example.com'}).add(Post).generate
 
+With the Rails-specific class, you could even get away with as little code as:
+
+  BigSitemapRails.new.add(Post).generate
+
 === Pinging Search Engines
 
 To ping search engines, call <code>ping_search_engines</code> after you generate the sitemap:
 
-  sitemap.generate
-  sitemap.ping_search_engines
+  sitemap.generate.ping_search_engines
+
+=== Location URLs
+
+By default, URLs for the "loc" values are generated in the form:
+
+  :base_url/:path|<table_name>/<to_param>|<id>
+
+Alternatively, you can pass a lambda.  For example, to make use of your Rails route helper:
+
+  sitemap.add(Post,
+    :location => lambda { |post| post_url(post) }
+  )
 
 === Change Frequency, Priority and Last Modified
 
 You can control "changefreq", "priority" and "lastmod" values for each record individually by passing lambdas instead of fixed values:
 
-  sitemap.add(Posts,
-    :change_frequency => lambda {|post| ... },
-    :priority         => lambda {|post| ... },
-    :last_modified    => lambda {|post| ... }
+  sitemap.add(Post,
+    :change_frequency => lambda { |post| ... },
+    :priority         => lambda { |post| ... },
+    :last_modified    => lambda { |post| ... }
   )
 
 === Find Methods
@@ -87,7 +116,28 @@ Your models must provide either a <code>find_for_sitemap</code> or <code>all</co
 
 Additionally, you models must provide a <code>count_for_sitemap</code> or <code>count</code> class method that returns a count of the instances to be included.
 
-If you're using ActiveRecord (Rails) or DataMapper then <code>all</code> and <code>count</code> are already provided and you don't need to do anything unless you want to include a subset of records.  If you provide your own <code>find_for_sitemap</code> or <code>all</code> method then it should be able to handle the <code>:offset</code> and <code>:limit</code> options, in the same way that ActiveRecord and DataMapper handle them.  This is especially important if you have more than 50,000 URLs.
+If you're using ActiveRecord (Rails) or DataMapper then <code>all</code> and <code>count</code> are already provided and you can make use of any supported parameter: (:conditions, :limit, :joins, :select, :order, :include, :group)
+
+  sitemap.add(Track,
+    :select     => "id, permalink, user_id, updated_at",
+    :include    => :user,
+    :conditions => "public = 1 AND state = 'finished' AND user_id IS NOT NULL",
+    :order      => "id ASC"
+  )
+
+If you provide your own <code>find_for_sitemap</code> or <code>all</code> method then it should be able to handle the <code>:offset</code> and <code>:limit</code> options, in the same way that ActiveRecord and DataMapper handle them.  This is especially important if you have more than 50,000 URLs.
+
+=== Partial Update
+
+If you enable <code>:partial_update</code>, the filename will include an id smaller than the id of the first entry. This is perfect to update just the last file with new entries without the need to re-generate files being already there.
+
+=== Lock Generation Process
+
+To prevent another process overwriting from the generated files, use the <code>with_lock</code> method:
+
+  sitemap.with_lock do
+    sitemap.generate
+  end
 
 === Cleaning the Sitemaps Directory
 
@@ -95,23 +145,23 @@ Calling the <code>clean</code> method will remove all files from the Sitemaps di
 
 == Limitations
 
-If your database is likely to shrink during the time it takes to create the sitemap then you might run into problems (the final, batched SQL select will overrun by setting a limit that is too large since it is calculated from the count, which is queried at the very beginning).  Patches welcome!
+If your database is likely to shrink during the time it takes to create the sitemap then you might run into problems (the final, batched SQL select will overrun by setting a limit that is too large since it is calculated from the count, which is queried at the very beginning).  In this case and your database uses incremental primary IDs then you might want to use the <code>:partial_update</code> option, which looks at the last ID instead of paginating.
 
 == TODO
 
-Tests for Rails components.
+Tests for framework-specific components.
 
 == Credits
 
 Thanks to Alastair Brunton and Harry Love, who's work provided a starting point for this library.
-http://scoop.cheerfactory.co.uk/2008/02/26/google-sitemap-generator/
 
-Thanks to those who have contributed patches:
+Thanks also to those who have contributed patches:
 
 * Mislav Marohnić
 * Jeff Schoolcraft
 * Dalibor Nasevic
+* Tobias Bielohlawek (http://www.rngtng.com)
 
 == Copyright
 
-Copyright (c) 2009 Stateless Systems (http://statelesssystems.com). See LICENSE for details.
+Copyright (c) 2010 Stateless Systems (http://statelesssystems.com). See LICENSE for details.

data/Rakefile

@@ -0,0 +1,52 @@
+require 'rake'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |s|
+    s.name = "big_sitemap"
+    s.summary = %Q{A Sitemap generator specifically designed for large sites (although it works equally well with small sites)}
+    s.email = %w(tobi@soundcloud.com alexrabarts@gmail.com)
+    s.homepage = "http://github.com/rngtng/big_sitemap"
+    s.description = "A Sitemap generator specifically designed for large sites (although it works equally well with small sites)"
+    s.authors = ["Tobias Bielohlawek", "Alex Rabarts"]
+    s.add_dependency 'bundler'
+  end
+rescue LoadError
+  puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
+end
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = 'big_sitemap'
+  rdoc.options << '--line-numbers' << '--inline-source'
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |t|
+  t.libs << 'lib' << 'test' << Rake.original_dir
+  t.pattern = 'test/**/*_test.rb'
+  t.verbose = false
+end
+
+begin
+  require 'rcov/rcovtask'
+  Rcov::RcovTask.new do |t|
+    t.libs << 'test'
+    t.test_files = FileList['test/**/*_test.rb']
+    t.verbose = true
+  end
+rescue LoadError
+  puts "RCov is not available. In order to run rcov, you must: sudo gem install spicycode-rcov"
+end
+
+begin
+  require 'cucumber/rake/task'
+  Cucumber::Rake::Task.new(:features)
+rescue LoadError
+  puts "Cucumber is not available. In order to run features, you must: sudo gem install cucumber"
+end
+
+task :default => :test

data/VERSION.yml

@@ -1,4 +1,4 @@
 --- 
 :patch: 1
 :major: 0
-:minor: 5
+:minor: 8

data/big_sitemap.gemspec

@@ -0,0 +1,58 @@
+# Generated by jeweler
+# DO NOT EDIT THIS FILE
+# Instead, edit Jeweler::Tasks in Rakefile, and run `rake gemspec`
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+  s.name = %q{big_sitemap}
+  s.version = "0.8.1"
+
+  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+  s.authors = ["Tobias Bielohlawek", "Alex Rabarts"]
+  s.date = %q{2011-01-25}
+  s.description = %q{A Sitemap generator specifically designed for large sites (although it works equally well with small sites)}
+  s.email = ["tobi@soundcloud.com", "alexrabarts@gmail.com"]
+  s.extra_rdoc_files = [
+    "LICENSE",
+     "README.rdoc"
+  ]
+  s.files = [
+    ".gitignore",
+     "Gemfile",
+     "Gemfile.lock",
+     "History.txt",
+     "LICENSE",
+     "README.rdoc",
+     "Rakefile",
+     "VERSION.yml",
+     "big_sitemap.gemspec",
+     "lib/big_sitemap.rb",
+     "lib/big_sitemap/builder.rb",
+     "test/big_sitemap_test.rb",
+     "test/fixtures/test_model.rb",
+     "test/test_helper.rb"
+  ]
+  s.homepage = %q{http://github.com/rngtng/big_sitemap}
+  s.rdoc_options = ["--charset=UTF-8"]
+  s.require_paths = ["lib"]
+  s.rubygems_version = %q{1.3.7}
+  s.summary = %q{A Sitemap generator specifically designed for large sites (although it works equally well with small sites)}
+  s.test_files = [
+    "test/big_sitemap_test.rb",
+     "test/fixtures/test_model.rb",
+     "test/test_helper.rb"
+  ]
+
+  if s.respond_to? :specification_version then
+    current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
+    s.specification_version = 3
+
+    if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
+      s.add_runtime_dependency(%q<bundler>, [">= 0"])
+    else
+      s.add_dependency(%q<bundler>, [">= 0"])
+    end
+  else
+    s.add_dependency(%q<bundler>, [">= 0"])
+  end
+end

data/lib/big_sitemap/builder.rb

@@ -1,104 +1,96 @@
-require 'builder'
+require 'fileutils'
 require 'zlib'
 
 class BigSitemap
-  class Builder < Builder::XmlMarkup
-    NAMESPACE = 'http://www.sitemaps.org/schemas/sitemap/0.9'
+  class Builder
     MAX_URLS = 50000
+    HEADER_ATTRIBUTES = {
+      'xmlns' => 'http://www.sitemaps.org/schemas/sitemap/0.9',
+      'xmlns:xsi' => "http://www.w3.org/2001/XMLSchema-instance",
+      'xsi:schemaLocation' => "http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd" 
+    }
 
     def initialize(options)
-      @gzip = options.delete(:gzip)
-      @max_urls = options.delete(:max_urls) || MAX_URLS
-      @index = options.delete(:index)
-      @paths = []
-      @parts = 0
-
-      if @filename = options.delete(:filename)
-        options[:target] = _get_writer
-      end
-
-      super(options)
+      @gzip           = options.delete(:gzip)
+      @max_urls       = options.delete(:max_urls) || MAX_URLS
+      @type           = options.delete(:type)
+      @paths          = []
+      @parts          = options.delete(:start_part_id) || 0
+      @custom_part_nr = options.delete(:partial_update)
+
+      @filename = options.delete(:filename)
+      @current_filename = nil
+      @tmp_filename     = nil
+      @target = _get_writer
 
+      @level = 0
       @opened_tags = []
       _init_document
     end
 
-    def add_url!(url, time = nil, frequency = nil, priority = nil)
-      _rotate if @max_urls == @urls
+    def add_url!(url, time = nil, frequency = nil, priority = nil, part_nr = nil)
+      _rotate(part_nr) if @max_urls == @urls
+
+      _open_tag 'url'
+      tag! 'loc', url
+      tag! 'lastmod', time.utc.strftime('%Y-%m-%dT%H:%M:%S+00:00') if time
+      tag! 'changefreq', frequency if frequency
+      tag! 'priority', priority if priority
+      _close_tag 'url'
 
-      tag!(@index ? 'sitemap' : 'url') do
-        loc url
-        # W3C format is the subset of ISO 8601
-        lastmod(time.utc.strftime('%Y-%m-%dT%H:%M:%S+00:00')) unless time.nil?
-        changefreq(frequency) unless frequency.nil?
-        priority(priority) unless priority.nil?
-      end
       @urls += 1
     end
 
+    def paths!
+      @paths
+    end
+
     def close!
       _close_document
       target!.close if target!.respond_to?(:close)
+      File.delete(@current_filename) if File.exists?(@current_filename)
+      File.rename(@tmp_filename, @current_filename)
     end
 
-    def paths!
-      @paths
+    def target!
+      @target
     end
 
     private
 
     def _get_writer
-      if @filename
-        filename = @filename.dup
-        filename << "_#{@parts}" if @parts > 0
-        filename << '.xml'
-        filename << '.gz' if @gzip
-        _open_writer(filename)
-      else
-        target!
-      end
+      filename = @filename.dup
+      filename << "_#{@parts}" if @parts > 0
+      filename << '.xml'
+      filename << '.gz' if @gzip
+      _open_writer(filename)
     end
 
     def _open_writer(filename)
-      file = File.open(filename, 'w+')
+      @current_filename = filename
+      @tmp_filename     = filename + ".tmp"
       @paths << filename
-      @gzip ? Zlib::GzipWriter.new(file) : file
+      file = ::File.open(@tmp_filename, 'w+')
+      @gzip ? ::Zlib::GzipWriter.new(file) : file
     end
 
-    def _init_document
+    def _init_document( name = 'urlset', attrs = HEADER_ATTRIBUTES)
       @urls = 0
-      instruct!
-      _open_tag(@index ? 'sitemapindex' : 'urlset', :xmlns => NAMESPACE)
+      target!.print '<?xml version="1.0" encoding="UTF-8"?>'
+      _newline
+      _open_tag name, attrs
     end
 
-    def _rotate
+    def _rotate(part_nr = nil)
       # write out the current document and start writing into a new file
       close!
-      @parts += 1
+      @parts = (part_nr && @custom_part_nr) ? part_nr : @parts + 1
       @target = _get_writer
       _init_document
     end
 
-    # add support for:
-    #   xml.open_foo!(attrs)
-    #   xml.close_foo!
-    def method_missing(method, *args, &block)
-      if method.to_s =~ /^(open|close)_(.+)!$/
-        operation, name = $1, $2
-        name = "#{name}:#{args.shift}" if Symbol === args.first
-
-        if 'open' == operation
-          _open_tag(name, args.first)
-        else
-          _close_tag(name)
-        end
-      else
-        super
-      end
-    end
-
     # opens a tag, bumps up level but doesn't require a block
-    def _open_tag(name, attrs)
+    def _open_tag(name, attrs = {})
       _indent
       _start_tag(name, attrs)
       _newline
@@ -106,6 +98,23 @@ class BigSitemap
       @opened_tags << name
     end
 
+    def _start_tag(name, attrs = {})
+      attrs = attrs.map { |attr,value| %Q( #{attr}="#{value}") }.join('')
+      target!.print "<#{name}#{attrs}>"
+    end
+
+    def tag!(name, content, attrs = {})
+      _indent
+      _start_tag(name, attrs)
+      target!.print content.to_s.gsub('&', '&amp;')
+      _end_tag(name)
+      _newline
+    end
+
+    def _end_tag(name)
+      target!.print "</#{name}>"
+    end
+
     # closes a tag block by decreasing the level and inserting a close tag
     def _close_tag(name)
       @opened_tags.pop
@@ -120,5 +129,40 @@ class BigSitemap
         _close_tag(name)
       end
     end
+
+    def _indent
+      return if @gzip
+      target!.print "  " * @level
+    end
+
+    def _newline
+      return if @gzip
+      target!.puts ''
+    end
   end
+
+  class IndexBuilder < Builder
+    def _init_document(name = 'sitemapindex', attrs = {'xmlns' => 'http://www.sitemaps.org/schemas/sitemap/0.9'})
+      attrs.merge('xmlns:geo' => "http://www.google.com/geo/schemas/sitemap/1.0")
+      super(name, attrs)
+    end
+
+    def add_url!(url, time = nil)
+      _open_tag 'sitemap'
+      tag! 'loc', url
+      tag! 'lastmod', time.utc.strftime('%Y-%m-%dT%H:%M:%S+00:00') if time
+      _close_tag 'sitemap'
+    end
+  end
+
+  class GeoBuilder < Builder
+    #_build_geo if @geo
+
+    # def _build_geo
+    #   geo :geo do
+    #     geo :format, 'kml'
+    #   end
+    # end
+  end
+
 end

data/lib/big_sitemap.rb

@@ -1,7 +1,7 @@
 require 'uri'
+require 'fileutils'
+
 require 'big_sitemap/builder'
-require 'extlib'
-require 'action_controller' if defined? Rails
 
 class BigSitemap
   DEFAULTS = {
@@ -22,13 +22,10 @@ class BigSitemap
   TIMESTAMP_METHODS = [:updated_at, :updated_on, :updated, :created_at, :created_on, :created]
   PARAM_METHODS     = [:to_param, :id]
 
-  include ActionController::UrlWriter if defined? Rails
-
   def initialize(options)
     @options = DEFAULTS.merge options
 
-    # Use Rails' default_url_options if available
-    @default_url_options = defined?(Rails) ? default_url_options : {}
+    @default_url_options = options.delete(:default_url_options) || {}
 
     if @options[:max_per_sitemap] <= 1
       raise ArgumentError, '":max_per_sitemap" must be greater than 1'
@@ -49,13 +46,7 @@ class BigSitemap
       raise ArgumentError, '":batch_size" must be less than ":max_per_sitemap"'
     end
 
-    @options[:document_root] ||= begin
-      if defined? Rails
-        "#{Rails.root}/public"
-      elsif defined? Merb
-        "#{Merb.root}/public"
-      end
-    end
+    @options[:document_root] ||= document_root
 
     unless @options[:document_root]
       raise ArgumentError, 'Document root must be specified with the ":document_root" option'
@@ -69,27 +60,77 @@ class BigSitemap
   end
 
   def add(model, options={})
-    options[:path] ||= Extlib::Inflection.tableize(model.to_s)
+    options[:path]           ||= table_name(model)
+    options[:filename]       ||= file_name(model)
+    options[:primary_column] ||= 'id' if model.new.respond_to?('id')
+    options[:partial_update]   = @options[:partial_update] && options[:partial_update] != false
     @sources << [model, options.dup]
-    return self
+    self
+  end
+
+  def add_static(url, time = nil, frequency = nil, priority = nil)
+    @static_pages ||= []
+    @static_pages << [url, time, frequency, priority]
+    self
+  end
+
+  def with_lock
+    lock!
+    begin
+      yield
+    ensure
+      unlock!
+    end
+  rescue Errno::EACCES => e
+    STDERR.puts "Lockfile exists"
+  end
+
+  def table_name(model)
+    model.table_name
+  end
+
+  def file_name(name)
+    name = table_name(name) unless name.is_a? String
+    "#{@file_path}/sitemap_#{name}"
+  end
+
+  def document_root
   end
 
   def clean
     Dir["#{@file_path}/sitemap_*.{xml,xml.gz}"].each do |file|
       FileUtils.rm file
     end
-    return self
+    self
   end
 
   def generate
+    prepare_update
+
+    generate_models
+    generate_static
+    generate_sitemap_index
+    self
+  end
+
+  def generate_models
     for model, options in @sources
-      with_sitemap(Extlib::Inflection::tableize(model.to_s)) do |sitemap|
+      with_sitemap(model, options.dup) do |sitemap|
+        last_id = nil #id of last processed item
         count_method = pick_method(model, COUNT_METHODS)
         find_method  = pick_method(model, FIND_METHODS)
         raise ArgumentError, "#{model} must provide a count_for_sitemap class method" if count_method.nil?
         raise ArgumentError, "#{model} must provide a find_for_sitemap class method" if find_method.nil?
 
-        count        = model.send(count_method)
+        find_options = {}
+        [:conditions, :limit, :joins, :select, :order, :include, :group].each do |key|
+          find_options[key] = options.delete(key)
+        end
+
+        primary_column   = options.delete(:primary_column)
+
+        count = model.send(count_method, find_options.merge(:select => (primary_column || '*'), :include => nil))
+        count = find_options[:limit].to_i if find_options[:limit] && find_options[:limit].to_i < count
         num_sitemaps = 1
         num_batches  = 1
 
@@ -99,18 +140,22 @@ class BigSitemap
         end
         batches_per_sitemap = num_batches.to_f / num_sitemaps.to_f
 
-        find_options = options.except(:path, :num_items, :priority, :change_frequency, :last_modified)
-
         for sitemap_num in 1..num_sitemaps
           # Work out the start and end batch numbers for this sitemap
           batch_num_start = sitemap_num == 1 ? 1 : ((sitemap_num * batches_per_sitemap).ceil - batches_per_sitemap + 1).to_i
           batch_num_end   = (batch_num_start + [batches_per_sitemap, num_batches].min).floor - 1
 
           for batch_num in batch_num_start..batch_num_end
-            offset       = ((batch_num - 1) * @options[:batch_size])
-            limit        = (count - offset) < @options[:batch_size] ? (count - offset - 1) : @options[:batch_size]
+            offset        = (batch_num - 1) * @options[:batch_size]
+            limit         = (count - offset) < @options[:batch_size] ? (count - offset) : @options[:batch_size]
             find_options.update(:limit => limit, :offset => offset) if num_batches > 1
 
+            if last_id && primary_column
+              find_options.update(:limit => limit, :offset => nil)
+              primary_column_value = last_id.to_s.gsub("'", %q(\\\')) #escape '
+              find_options.update(:conditions => [find_options[:conditions], "(#{primary_column} > '#{primary_column_value}')"].compact.join(' AND '))
+            end
+
             model.send(find_method, find_options).each do |record|
               last_mod = options[:last_modified]
               if last_mod.is_a?(Proc)
@@ -122,8 +167,12 @@ class BigSitemap
 
               param_method = pick_method(record, PARAM_METHODS)
 
-              location = defined?(Rails) ? polymorphic_url(record) : nil rescue nil
-              location ||= "#{root_url}/#{strip_leading_slash(options[:path])}/#{record.send(param_method)}"
+              location = options[:location]
+              if location.is_a?(Proc)
+                location = location.call(record)
+              else
+                location = "#{root_url}/#{strip_leading_slash(options[:path])}/#{record.send(param_method)}"
+              end
 
               change_frequency = options[:change_frequency] || 'weekly'
               freq = change_frequency.is_a?(Proc) ? change_frequency.call(record) : change_frequency
@@ -131,16 +180,36 @@ class BigSitemap
               priority = options[:priority]
               pri = priority.is_a?(Proc) ? priority.call(record) : priority
 
-              sitemap.add_url!(location, last_mod, freq, pri)
+              last_id = primary_column ? record.send(primary_column) : nil
+              sitemap.add_url!(location, last_mod, freq, pri, last_id)
             end
           end
         end
       end
     end
+    self
+  end
 
-    generate_sitemap_index
+  def generate_static
+    return self if Array(@static_pages).empty?
+    with_sitemap('static', :type => 'static') do |sitemap|
+      @static_pages.each do |location, last_mod, freq, pri|
+        sitemap.add_url!(location, last_mod, freq, pri)
+      end
+    end
+    self
+  end
 
-    return self
+  # Create a sitemap index document
+  def generate_sitemap_index(files = nil)
+    files ||= Dir["#{@file_path}/sitemap_*.{xml,xml.gz}"]
+    with_sitemap 'index', :type => 'index' do |sitemap|
+      for path in files
+        next if path =~ /index/
+        sitemap.add_url!(url_for_sitemap(path), File.stat(path).mtime)
+      end
+    end
+    self
   end
 
   def ping_search_engines
@@ -186,16 +255,40 @@ class BigSitemap
 
   private
 
-  def with_sitemap(name, options={})
-    options[:index] = name == 'index'
-    options[:filename] = "#{@file_path}/sitemap_#{name}"
-    options[:max_urls] = @options[:max_per_sitemap]
-
-    unless options[:gzip] = @options[:gzip]
-      options[:indent] = 2
+  def prepare_update
+    @files_to_move = []
+    @sources.each do |model, options|
+      if options[:partial_update] && primary_column = options[:primary_column] && last_id = get_last_id(options[:filename])
+        primary_column_value       = last_id.to_s.gsub("'", %q(\\\')) #escape '
+        options[:conditions]       = [options[:conditions], "(#{primary_column} >= '#{primary_column_value}')"].compact.join(' AND ')
+        options[:start_part_id]    = last_id
+      end
     end
+  end
+
+  def lock!(lock_file = 'generator.lock')
+    File.open("#{@file_path}/#{lock_file}", 'w', File::EXCL)
+  end
 
-    sitemap = Builder.new(options)
+  def unlock!(lock_file = 'generator.lock')
+    FileUtils.rm "#{@file_path}/#{lock_file}"
+  end
+
+  def with_sitemap(name, options={})
+    options[:filename] ||= file_name(name)
+    options[:type]     ||= 'sitemap'
+    options[:max_urls] ||= @options["max_per_#{options[:type]}".to_sym]
+    options[:gzip]     ||= @options[:gzip]
+    options[:indent]     = options[:gzip] ? 0 : 2
+
+    sitemap = if options[:type] == 'index'
+      IndexBuilder.new(options)
+    elsif options[:geo]
+      options[:filename] << '_kml'
+      GeoBuilder.new(options)
+    else
+      Builder.new(options)
+    end
 
     begin
       yield sitemap
@@ -209,6 +302,12 @@ class BigSitemap
     str.sub(/^\//, '')
   end
 
+  def get_last_id(filename)
+    Dir["#{filename}*.{xml,xml.gz}"].map do |file|
+      file.to_s.scan(/#{filename}_(.+).xml/).flatten.last.to_i
+    end.sort.last
+  end
+
   def pick_method(model, candidates)
     method = nil
     candidates.each do |candidate|
@@ -221,19 +320,43 @@ class BigSitemap
   end
 
   def url_for_sitemap(path)
-    if @options[:path].blank?
-      "#{root_url}/#{File.basename(path)}"
-    else
-      "#{root_url}/#{@options[:path]}/#{File.basename(path)}"
-    end
+    [root_url, @options[:path], File.basename(path)].compact.join('/')
   end
 
-  # Create a sitemap index document
-  def generate_sitemap_index
-    with_sitemap 'index' do |sitemap|
-      for path in @sitemap_files
-        sitemap.add_url!(url_for_sitemap(path), File.stat(path).mtime)
-      end
-    end
+end
+
+
+
+class BigSitemapRails < BigSitemap
+
+  include ActionController::UrlWriter if defined? Rails
+
+  def initialize(options)
+    require 'action_controller'
+
+    super options.merge(:default_url_options => default_url_options)
+  end
+
+  def document_root
+    "#{Rails.root}/public"
   end
+end
+
+
+
+class BigSitemapMerb < BigSitemap
+
+  def initialize(options)
+    require 'extlib'
+    super
+  end
+
+  def document_root
+    "#{Merb.root}/public"
+  end
+
+  def table_name(model)
+    Extlib::Inflection.tableize(model.to_s)
+  end
+
 end

data/test/big_sitemap_test.rb

@@ -161,6 +161,7 @@ class BigSitemapTest < Test::Unit::TestCase
 
     should 'strip leading slashes from controller paths' do
       create_sitemap
+      add_model
       @sitemap.add(TestModel, :path => '/test_controller').generate
       assert(
         !elements(first_sitemaps_model_file, 'loc').first.text.match(/\/\/test_controller\//),
@@ -181,6 +182,29 @@ class BigSitemapTest < Test::Unit::TestCase
     end
   end
 
+  context 'add static method' do
+    should 'should generate static content' do
+      create_sitemap
+      @sitemap.add_static('/', Time.now, 'weekly', 0.5)
+      @sitemap.add_static('/about', Time.now, 'weekly', 0.5)
+      @sitemap.generate_static
+      elems = elements(static_sitemaps_file, 'loc')
+      assert_equal "/", elems.first.text
+      assert_equal "/about", elems.last.text
+    end
+  end
+
+  context 'sanatize XML chars' do
+    should 'should transform ampersands' do
+      create_sitemap
+      @sitemap.add_static('/something&else', Time.now, 'weekly', 0.5)
+      @sitemap.generate_static
+      elems = elements(static_sitemaps_file, 'loc')
+      assert Zlib::GzipReader.open(static_sitemaps_file).read.include?("/something&amp;else")
+      assert_equal "/something&else", elems.first.text
+    end
+  end
+
   context 'clean method' do
     should 'be chainable' do
       create_sitemap
@@ -202,6 +226,164 @@ class BigSitemapTest < Test::Unit::TestCase
     end
   end
 
+  context 'sitemap index' do
+    should 'generate for all xml files in directory' do
+      create_sitemap
+      @sitemap.clean
+      File.open("#{sitemaps_dir}/sitemap_file1.xml", 'w')
+      File.open("#{sitemaps_dir}/sitemap_file2.xml.gz", 'w')
+      File.open("#{sitemaps_dir}/sitemap_file3.txt", 'w')
+      File.open("#{sitemaps_dir}/file4.xml", 'w')
+      File.open(unzipped_sitemaps_index_file, 'w')
+      @sitemap.send :generate_sitemap_index
+
+      elem = elements(sitemaps_index_file, 'loc')
+      assert_equal 2, elem.size #no index and file3 and file4 found
+      assert_equal "http://example.com/sitemaps/sitemap_file1.xml", elem.first.text
+      assert_equal "http://example.com/sitemaps/sitemap_file2.xml.gz", elem.last.text
+    end
+
+    should 'generate for all for given file' do
+      create_sitemap
+      @sitemap.clean
+      File.open("#{sitemaps_dir}/sitemap_file1.xml", 'w')
+      File.open("#{sitemaps_dir}/sitemap_file2.xml.gz", 'w')
+      files = ["#{sitemaps_dir}/sitemap_file1.xml", "#{sitemaps_dir}/sitemap_file2.xml.gz"]
+      @sitemap.send :generate_sitemap_index, files
+
+      elem = elements(sitemaps_index_file, 'loc')
+      assert_equal 2, elem.size
+      assert_equal "http://example.com/sitemaps/sitemap_file1.xml", elem.first.text
+      assert_equal "http://example.com/sitemaps/sitemap_file2.xml.gz", elem.last.text
+    end
+  end
+
+  context 'get_last_id' do
+    should 'return last id' do
+      create_sitemap.clean
+      filename = "#{sitemaps_dir}/sitemap_file"
+      File.open("#{filename}_1.xml", 'w')
+      File.open("#{filename}_23.xml", 'w')
+      File.open("#{filename}_42.xml.gz", 'w')
+      File.open("#{filename}_9.xml", 'w')
+      assert_equal 42, @sitemap.send(:get_last_id, filename)
+    end
+
+    should 'return nil' do
+      create_sitemap.clean
+      filename = "#{sitemaps_dir}/sitemap_file"
+      assert_equal nil, @sitemap.send(:get_last_id, filename)
+    end
+  end
+
+  context 'partial update' do
+    should 'generate for all xml files in directory and delete last file' do
+      TestModel.current_id = last_id = 27
+      filename = "#{sitemaps_dir}/sitemap_test_models"
+
+      create_sitemap(:partial_update => true, :gzip => false, :batch_size => 5, :max_per_sitemap => 5, :max_per_index => 100).clean
+      add_model(:num_items => 50 - last_id) #TestModel
+
+      File.open("#{filename}.xml", 'w')
+      File.open("#{filename}_5.xml", 'w')
+      File.open("#{filename}_9.xml", 'w')
+      File.open("#{filename}_23.xml", 'w')
+      File.open("#{filename}_#{last_id}.xml", 'w')
+      @sitemap.generate
+
+      # Dir["#{sitemaps_dir}/*"].each do |d| puts d; end
+
+      assert File.exists?("#{filename}_48.xml")
+      assert File.exists?("#{filename}_#{last_id}.xml")
+      elems = elements("#{filename}_#{last_id}.xml", 'loc').map(&:text)
+
+      assert_equal 5, elems.size
+      (28..32).each do |i|
+        assert elems.include? "http://example.com/test_models/#{i}"
+      end
+
+      elems = elements(unzipped_sitemaps_index_file, 'loc').map(&:text)
+      assert elems.include? "http://example.com/sitemaps/sitemap_test_models.xml"
+      assert elems.include? "http://example.com/sitemaps/sitemap_test_models_9.xml"
+      assert elems.include? "http://example.com/sitemaps/sitemap_test_models_#{last_id}.xml"
+      assert elems.include? "http://example.com/sitemaps/sitemap_test_models_48.xml"
+    end
+
+    should 'generate sitemap, update should respect old files' do
+      max_id = 23
+      TestModel.current_id = 0
+      filename = "#{sitemaps_dir}/sitemap_test_models"
+
+      create_sitemap(:partial_update => true, :gzip => false, :batch_size => 5, :max_per_sitemap => 5, :max_per_index => 100).clean
+      add_model(:num_items => max_id) #TestModel
+      @sitemap.generate
+
+      # Dir["#{sitemaps_dir}/*"].each do |d| puts d; end
+
+      assert_equal 5, elements("#{filename}.xml", 'loc').size
+      assert_equal 5, elements("#{filename}_6.xml", 'loc').size
+      assert_equal 3, elements("#{filename}_21.xml", 'loc').size
+
+      TestModel.current_id = 20 #last_id is 21, so start with one below
+      create_sitemap(:partial_update => true, :gzip => false, :batch_size => 5, :max_per_sitemap => 5, :max_per_index => 100)
+      add_model( :num_items => 48 - TestModel.current_id ) #TestModel
+      @sitemap.generate
+
+      assert_equal 5, elements("#{filename}_6.xml", 'loc').size
+      assert_equal 5, elements("#{filename}_21.xml", 'loc').size
+
+      # Dir["#{sitemaps_dir}/*"].each do |d| puts d; end
+
+      elems = elements("#{filename}_26.xml", 'loc').map(&:text)
+      (26..30).each do |i|
+         assert elems.include? "http://example.com/test_models/#{i}"
+      end
+
+      #puts `cat /tmp/sitemaps/sitemap_test_models_41.xml`
+
+      assert_equal 3, elements("#{filename}_46.xml", 'loc').size
+    end
+
+    context 'lockfile' do
+      should 'create and delete lock file' do
+        sitemap = BigSitemap.new(:base_url => 'http://example.com', :document_root => tmp_dir)
+
+        sitemap.with_lock do
+          assert File.exists?('/tmp/sitemaps/generator.lock')
+        end
+
+        assert !File.exists?('/tmp/sitemaps/generator.lock')
+      end
+
+      should 'not catch error not related to lock' do
+        sitemap = BigSitemap.new(:base_url => 'http://example.com', :document_root => tmp_dir)
+
+        assert_raise RuntimeError do
+          sitemap.with_lock do
+            raise "Wrong"
+          end
+        end
+
+      end
+
+      should 'throw error if lock exits' do
+        sitemap = BigSitemap.new(:base_url => 'http://example.com', :document_root => tmp_dir)
+
+        sitemap.with_lock do
+          sitemap2 = BigSitemap.new(:base_url => 'http://example.com', :document_root => tmp_dir)
+
+          assert_nothing_raised do
+            sitemap2.with_lock do
+              raise "Should not be called"
+            end
+          end
+
+        end
+      end
+
+    end
+  end
+
   private
     def delete_tmp_files
       FileUtils.rm_rf(sitemaps_dir)
@@ -211,7 +393,7 @@ class BigSitemapTest < Test::Unit::TestCase
       @sitemap = BigSitemap.new({
         :base_url      => 'http://example.com',
         :document_root => tmp_dir,
-        :update_google => false
+        :ping_google => false
       }.update(options))
     end
 
@@ -239,7 +421,7 @@ class BigSitemapTest < Test::Unit::TestCase
 
     def add_model(options={})
       num_items = options.delete(:num_items) || default_num_items
-      TestModel.stubs(:num_items).returns(num_items)
+      TestModel.stubs(:count_for_sitemap).returns(num_items)
       @sitemap.add(TestModel, options)
     end
 
@@ -263,6 +445,10 @@ class BigSitemapTest < Test::Unit::TestCase
       "#{sitemaps_dir}/sitemap_test_models.xml.gz"
     end
 
+    def static_sitemaps_file
+      "#{sitemaps_dir}/sitemap_static.xml.gz"
+    end
+
     def second_sitemaps_model_file
       "#{sitemaps_dir}/sitemap_test_models_1.xml.gz"
     end
@@ -284,7 +470,8 @@ class BigSitemapTest < Test::Unit::TestCase
     end
 
     def elements(filename, el)
-      data = Nokogiri::XML.parse(Zlib::GzipReader.open(filename).read)
+      file_class = filename.include?('.gz') ? Zlib::GzipReader : File
+      data = Nokogiri::XML.parse(file_class.open(filename).read)
       data.search("//s:#{el}", ns)
     end
 

data/test/fixtures/test_model.rb

@@ -1,6 +1,10 @@
 class TestModel
   def to_param
-    object_id
+    id #|| object_id
+  end
+
+  def id
+    @id ||= TestModel.current_id += 1
   end
 
   def change_frequency
@@ -16,6 +20,10 @@ class TestModel
   end
 
   class << self
+    def table_name
+      'test_models'
+    end
+
     def count_for_sitemap
       self.find_for_sitemap.size
     end
@@ -30,5 +38,11 @@ class TestModel
       num_times.times { instances.push(self.new) }
       instances
     end
+
+    attr_writer :current_id
+
+    def current_id
+      @current_id ||= 0
+    end
   end
 end

data/test/test_helper.rb

@@ -1,10 +1,11 @@
 require 'rubygems'
+require 'bundler/setup'
+
 require 'test/unit'
 require 'shoulda'
 require 'mocha'
 require 'test/fixtures/test_model'
 
-$LOAD_PATH.unshift(File.dirname(__FILE__))
 require 'big_sitemap'
 
 class Test::Unit::TestCase

metadata

@@ -1,81 +1,70 @@
 --- !ruby/object:Gem::Specification 
 name: big_sitemap
 version: !ruby/object:Gem::Version 
-  hash: 9
+  hash: 61
   prerelease: false
   segments: 
   - 0
-  - 5
+  - 8
   - 1
-  version: 0.5.1
+  version: 0.8.1
 platform: ruby
 authors: 
+- Tobias Bielohlawek
 - Alex Rabarts
 autorequire: 
 bindir: bin
 cert_chain: []
 
-date: 2009-09-07 00:00:00 +01:00
+date: 2011-01-25 00:00:00 +00:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
-  name: builder
+  name: bundler
   prerelease: false
   requirement: &id001 !ruby/object:Gem::Requirement 
     none: false
     requirements: 
     - - ">="
       - !ruby/object:Gem::Version 
-        hash: 15
-        segments: 
-        - 2
-        - 1
-        - 2
-        version: 2.1.2
-  type: :runtime
-  version_requirements: *id001
-- !ruby/object:Gem::Dependency 
-  name: extlib
-  prerelease: false
-  requirement: &id002 !ruby/object:Gem::Requirement 
-    none: false
-    requirements: 
-    - - ">="
-      - !ruby/object:Gem::Version 
-        hash: 41
+        hash: 3
         segments: 
         - 0
-        - 9
-        - 9
-        version: 0.9.9
+        version: "0"
   type: :runtime
-  version_requirements: *id002
+  version_requirements: *id001
 description: A Sitemap generator specifically designed for large sites (although it works equally well with small sites)
-email: alexrabarts@gmail.com
+email: 
+- tobi@soundcloud.com
+- alexrabarts@gmail.com
 executables: []
 
 extensions: []
 
 extra_rdoc_files: 
-- README.rdoc
 - LICENSE
+- README.rdoc
 files: 
+- .gitignore
+- Gemfile
+- Gemfile.lock
 - History.txt
+- LICENSE
 - README.rdoc
+- Rakefile
 - VERSION.yml
-- lib/big_sitemap/builder.rb
+- big_sitemap.gemspec
 - lib/big_sitemap.rb
+- lib/big_sitemap/builder.rb
 - test/big_sitemap_test.rb
 - test/fixtures/test_model.rb
 - test/test_helper.rb
-- LICENSE
 has_rdoc: true
-homepage: http://github.com/alexrabarts/big_sitemap
+homepage: http://github.com/rngtng/big_sitemap
 licenses: []
 
 post_install_message: 
 rdoc_options: 
-- --inline-source
 - --charset=UTF-8
 require_paths: 
 - lib
@@ -102,7 +91,9 @@ requirements: []
 rubyforge_project: 
 rubygems_version: 1.3.7
 signing_key: 
-specification_version: 2
+specification_version: 3
 summary: A Sitemap generator specifically designed for large sites (although it works equally well with small sites)
-test_files: []
-
+test_files: 
+- test/big_sitemap_test.rb
+- test/fixtures/test_model.rb
+- test/test_helper.rb