sitemap_generator 2.1.0 → 2.1.1

data/Gemfile

@@ -19,6 +19,6 @@ gem 'nokogiri', '1.4.4'
 gem 'sqlite3-ruby', '1.3.1', :require => 'sqlite3'
 
 group :test do
-  gem 'ruby-debug', '0.10.3'
-  gem 'ruby-debug-base', '0.10.3'
+  gem 'ruby-debug', '~>0.10'
+  gem 'ruby-debug-base', '~>0.10'
 end

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: ./
   specs:
-    sitemap_generator (2.1.0)
+    sitemap_generator (2.1.1)
 
 GEM
   remote: http://rubygems.org/
@@ -77,7 +77,7 @@ DEPENDENCIES
   rake (>= 0.8.7)
   rspec (= 1.3.1)
   rspec-rails (~> 1.3.2)
-  ruby-debug (= 0.10.3)
-  ruby-debug-base (= 0.10.3)
+  ruby-debug (~> 0.10)
+  ruby-debug-base (~> 0.10)
   sitemap_generator!
   sqlite3-ruby (= 1.3.1)

data/README.md

@@ -23,10 +23,10 @@ Does your website use SitemapGenerator to generate Sitemaps?  Where would you be
 
 <a href='http://www.pledgie.com/campaigns/15267'><img alt='Click here to lend your support to: SitemapGenerator and make a donation at www.pledgie.com !' src='http://pledgie.com/campaigns/15267.png?skin_name=chrome' border='0' /></a>
 
-
 Changelog
 -------
 
+- v2.1.1: Support calling `create()` multiple times in a sitemap config.  Support host names with path segments so you can use a `default_host` like `'http://mysite.com/subdirectory/'`.  Turn off `include_index` when the `sitemaps_host` differs from `default_host`.  Add docs about how to upload to remote hosts.
 - v2.1.0: [News sitemap][sitemap_news] support
 - v2.0.1.pre2: Fix uploading to the (bucket) root on a remote server
 - v2.0.1.pre1: Support read-only filesystems like Heroku by supporting uploading to remote host
@@ -155,6 +155,47 @@ To ensure that your application's sitemaps are available after a deployment you
           run "cd #{latest_release} && RAILS_ENV=#{rails_env} rake sitemap:refresh"
         end
 
+Upload Sitemaps to a Remote Host
+----------
+
+Sometimes it is desirable to host your sitemap files on a remote server and point robots
+and search engines to the remote files.  For example if you are using a host like Heroku
+which doesn't allow writing to the local filesystem.  You still require *some* write access
+because the sitemap files need to be written out before uploading, so generally a host will
+give you write access to a temporary directory.  On Heroku this is `tmp/` in your application
+directory.
+
+Sitemap Generator uses CarrierWave to support uploading to Amazon S3 store, Rackspace Cloud Files store, and MongoDB's GridF - whatever CarrierWave supports.
+
+1. Please see [this wiki page][remote_hosts] for more information about setting up CarrierWave, SitemapGenerator and Rails.
+
+2. Once you have CarrierWave setup and configured all you need to do is set some options in your sitemap config, such as:
+
+   * `default_host` - your website host name
+   * `sitemaps_host` - the remote host where your sitemaps will be hosted
+   * `public_path` - the directory to write sitemaps to locally e.g. `tmp/`
+   * `sitemaps_path` - set to a directory/path if you don't want to upload to the root of your `sitemaps_host`
+   * `adapter` - instance of `SitemapGenerator::WaveAdapter`
+
+   For Example:
+
+        SitemapGenerator::Sitemap.default_host = "http://www.example.com"
+        SitemapGenerator::Sitemap.sitemaps_host = "http://s3.amazonaws.com/sitemap-generator/"
+        SitemapGenerator::Sitemap.public_path = 'tmp/'
+        SitemapGenerator::Sitemap.sitemaps_path = 'sitemaps/'
+        SitemapGenerator::Sitemap.adapter = SitemapGenerator::WaveAdapter.new
+
+3. Update your `robots.txt` file to point robots to the remote sitemap index file, e.g:
+
+        Sitemap: http://s3.amazonaws.com/sitemap-generator/sitemaps/sitemap_index.xml.gz
+
+You generate your sitemaps as usual using `rake sitemap:refresh`.
+
+Note that SitemapGenerator will automatically turn off `include_index` in this case because
+the `sitemaps_host` does not match the `default_host`.  The link to the sitemap index file
+that would otherwise be included would point to a different host than the rest of the links
+in the sitemap, something that the sitemap rules forbid.
+
 Sitemap Configuration
 ======
 
@@ -344,13 +385,16 @@ The following options are supported:
 
 * `filename` - Symbol.  The **base name for the files** that will be generated.  The default value is `:sitemap`.  This yields sitemaps with names like `sitemap1.xml.gz`, `sitemap2.xml.gz`, `sitemap3.xml.gz` etc, and a sitemap index named `sitemap_index.xml.gz`.  If we now set the value to `:geo` the sitemaps would be named `geo1.xml.gz`, `geo2.xml.gz`, `geo3.xml.gz` etc, and the sitemap index would be named `geo_index.xml.gz`.
 
-* `include_index` - Boolean.  Whether to **add a link to the sitemap index** to the current sitemap.  This points search engines to your Sitemap Index to include it in the indexing of your site.  Default is `true`.
+* `include_index` - Boolean.  Whether to **add a link to the sitemap index** to the current sitemap.  This points search engines to your Sitemap Index to include it in the indexing of your site.  Default is `true`.  Turned off when `sitemaps_host` is set or within a `group()` block.
 
-* `include_root` - Boolean.  Whether to **add the root** url i.e. '/' to the current sitemap.  Default is `true`.
+* `include_root` - Boolean.  Whether to **add the root** url i.e. '/' to the current sitemap.  Default is `true`.  Turned off within a `group()` block.
 
 * `public_path` - String.  A **full or relative path** to the `public` directory or the directory you want to write sitemaps into.  Defaults to `public/` under your application root or relative to the current working directory.
 
-* `sitemaps_host` - String.  **Host including protocol** to use when generating a link to a sitemap file i.e. the hostname of the server where the sitemaps are hosted.  The value will differ from the hostname in your sitemap links.  For example: `'http://amazon.aws.com/'`
+* `sitemaps_host` - String.  **Host including protocol** to use when generating a link to a sitemap file i.e. the hostname of the server where the sitemaps are hosted.  The value will differ from the hostname in your sitemap links.  For example: `'http://amazon.aws.com/'`.  Note that `include_index` is
+automatically turned off when the `sitemaps_host` does not match `default_host`.
+Because the link to the sitemap index file that would otherwise be added would point to a
+different host than the rest of the links in the sitemap.  Something that the sitemap rules forbid.
 
 * `sitemaps_namer` - A `SitemapGenerator::SitemapNamer` instance **for generating sitemap names**.  You can read about Sitemap Namers by reading the API docs.  Sitemap Namers don't apply to the sitemap index.  You can only modify the name of the index file using the `filename` option.  Sitemap Namers allow you to set the name, extension and number sequence for sitemap files.
 
@@ -358,6 +402,12 @@ The following options are supported:
 
 * `verbose` - Boolean.  Whether to **output a sitemap summary** describing the sitemap files and giving statistics about your sitemap.  Default is `false`.  When using the Rake tasks `verbose` will be `true` unless you pass the `-s` option.
 
+* `adapter` - Instance.  The default adapter is a `SitemapGenerator::FileAdapter`
+  which simply writes files to the filesystem.  You can use a `SitemapGenerator::WaveAdapter`
+  for uploading sitemaps to remote servers - useful for read-only hosts such as Heroku.  Or
+  you can provide an instance of your own class to provide custom behavior.  Your class must
+  define a write method which takes a `SitemapGenerator::Location` and raw XML data.
+
 Sitemap Groups
 =======
 
@@ -578,3 +628,4 @@ Copyright (c) 2009 Karl Varga released under the MIT license
 [image_tags]:http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=178636
 [geo_tags]:http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=94555
 [news_tags]:http://www.google.com/support/news_pub/bin/answer.py?answer=74288
+[remote_hosts]:https://github.com/kjvarga/sitemap_generator/wiki/Generate-Sitemaps-on-read-only-filesystems-like-Heroku

data/VERSION

@@ -1 +1 @@
-2.1.0
+2.1.1

data/lib/sitemap_generator/builder/sitemap_url.rb

@@ -3,11 +3,27 @@ require 'uri'
 
 module SitemapGenerator
   module Builder
+    # A Hash-like class for holding information about a sitemap URL and
+    # generating an XML <url> element suitable for sitemaps.
     class SitemapUrl < Hash
 
-      # Call with:
-      #   sitemap - a Sitemap instance, or
-      #   path, options - a path for the URL and options hash
+      # Return a new instance with options configured on it.
+      #
+      # == Arguments
+      # * sitemap - a Sitemap instance, or
+      # * path, options - a path string and options hash
+      #
+      # == Options
+      # Requires a host to be set.  If passing a sitemap, the sitemap must have a +default_host+
+      # configured.  If calling with a path and options, you must include the <tt>:host</tt> option.
+      #
+      # * +priority+
+      # * +changefreq+
+      # * +lastmod+
+      # * +images+
+      # * +video+
+      # * +geo+
+      # * +news+
       def initialize(path, options={})
         if sitemap = path.is_a?(SitemapGenerator::Builder::SitemapFile) && path
           options.reverse_merge!(:host => sitemap.location.host, :lastmod => sitemap.lastmod)
@@ -16,17 +32,18 @@ module SitemapGenerator
 
         SitemapGenerator::Utilities.assert_valid_keys(options, :priority, :changefreq, :lastmod, :host, :images, :video, :geo, :news)
         options.reverse_merge!(:priority => 0.5, :changefreq => 'weekly', :lastmod => Time.now, :images => [], :news => {})
+        raise "Cannot generate a url without a host" unless options[:host].present?
         self.merge!(
-          :path => path,
-          :priority => options[:priority],
+          :path       => path,
+          :priority   => options[:priority],
           :changefreq => options[:changefreq],
-          :lastmod => options[:lastmod],
-          :host => options[:host],
-          :loc => URI.join(options[:host], path).to_s,
-          :images => prepare_images(options[:images], options[:host]),
-          :news => prepare_news(options[:news]),
-          :video => options[:video],
-          :geo => options[:geo]
+          :lastmod    => options[:lastmod],
+          :host       => options[:host],
+          :loc        => URI.join(options[:host], path.to_s.sub(/^\//, '')).to_s, # support host with subdirectory
+          :images     => prepare_images(options[:images], options[:host]),
+          :news       => prepare_news(options[:news]),
+          :video      => options[:video],
+          :geo        => options[:geo]
         )
       end
 
@@ -133,4 +150,4 @@ module SitemapGenerator
       end
     end
   end
-end
+end

data/lib/sitemap_generator/link_set.rb

@@ -10,8 +10,8 @@ module SitemapGenerator
     attr_reader :default_host, :sitemaps_path, :filename
     attr_accessor :verbose, :yahoo_app_id, :include_root, :include_index, :sitemaps_host, :adapter
 
-    # Add links to the link set by evaluating the block.  The block should
-    # contains calls to sitemap methods like:
+    # Create a new sitemap index and sitemap files.  Pass a block calls to the following
+    # methods:
     # * +add+   - Add a link to the current sitemap
     # * +group+ - Start a new group of sitemaps
     #
@@ -25,9 +25,12 @@ module SitemapGenerator
     # * <tt>:finalize</tt> - The sitemaps are written as they get full and at the end
     # of the block.  Pass +false+ as the value to prevent the sitemap or sitemap index
     # from being finalized.  Default is +true+.
+    #
+    # If you are calling +create+ more than once in your sitemap configuration file,
+    # make sure that you set a different +sitemaps_path+ or +filename+ for each call otherwise
+    # the sitemaps may be overwritten.
     def create(opts={}, &block)
-      @sitemap_index = nil if @sitemap_index && @sitemap_index.finalized? && !@protect_index
-      @sitemap = nil if @sitemap && @sitemap.finalized?
+      reset!
       set_options(opts)
       start_time = Time.now if @verbose
       interpreter.eval(:yield_sitemap => @yield_sitemap || SitemapGenerator.yield_sitemap?, &block)
@@ -47,8 +50,11 @@ module SitemapGenerator
     # Constructor
     #
     # == Options:
-    # * <tt>:adapter</tt> - subclass of SitemapGenerator::Adapter used for persisting the
-    #   sitemaps.  Default adapter is a SitemapGenerator::FileAdapter
+    # * <tt>:adapter</tt> - instance of a class with a write method which takes a SitemapGenerator::Location
+    #   and raw XML data and persists it.  The default adapter is a SitemapGenerator::FileAdapter
+    #   which simply writes files to the filesystem.  You can use a SitemapGenerator::WaveAdapter
+    #   for uploading sitemaps to remote servers - useful for read-only hosts such as Heroku.  Or
+    #   you can provide an instance of your own class to provide custom behavior.
     #
     # * <tt>:default_host</tt> - host including protocol to use in all sitemap links
     #   e.g. http://en.google.ca
@@ -57,8 +63,15 @@ module SitemapGenerator
     #   Defaults to the <tt>public/</tt> directory in your application root directory or
     #   the current working directory.
     #
-    # * <tt>:sitemaps_host</tt> - host (including protocol) to use in links to the sitemaps.  Useful if your sitemaps
-    #   are hosted o different server e.g. 'http://amazon.aws.com/'
+    # * <tt>:sitemaps_host</tt> - String.  <b>Host including protocol</b> to use when generating
+    #   a link to a sitemap file i.e. the hostname of the server where the sitemaps are hosted.
+    #   The value will differ from the hostname in your sitemap links.
+    #   For example: `'http://amazon.aws.com/'`.
+    #
+    #   Note that `include_index` is automatically turned off when the `sitemaps_host` does
+    #   not match `default_host`.  Because the link to the sitemap index file that would
+    #   otherwise be added would point to a different host than the rest of the links in
+    #   the sitemap.  Something that the sitemap rules forbid.
     #
     # * <tt>:sitemaps_path</tt> - path fragment within public to write sitemaps
     #   to e.g. 'en/'.  Sitemaps are written to <tt>public_path</tt> + <tt>sitemaps_path</tt>
@@ -69,11 +82,13 @@ module SitemapGenerator
     #
     # * <tt>:sitemaps_namer</tt> - A +SitemapNamer+ instance for generating the sitemap names.
     #
-    # * <tt>:include_root</tt> - whether to include the root url i.e. '/' in each group of sitemaps.
-    #   Default is true.
+    # * <tt>include_index</tt> - Boolean.  Whether to <b>add a link to the sitemap index<b>
+    #   to the current sitemap.  This points search engines to your Sitemap Index to
+    #   include it in the indexing of your site.  Default is `true`.  Turned off when
+    #  `sitemaps_host` is set or within a `group()` block.
     #
-    # * <tt>:include_index</tt> - whether to include the sitemap index URL in each group of sitemaps.
-    #   Default is true.
+    # * <tt>include_root</tt> - Boolean.  Whether to **add the root** url i.e. '/' to the
+    #   current sitemap.  Default is `true`.  Turned off within a `group()` block.
     #
     # * <tt>:verbose</tt> - If +true+, output a summary line for each sitemap and sitemap
     #   index that is created.  Default is +false+.
@@ -110,10 +125,10 @@ module SitemapGenerator
       retry
     end
 
-    # Create a new group of sitemaps.  Returns a new LinkSet instance with options set on it.
+    # Create a new group of sitemap files.
     #
-    # All groups share this LinkSet's sitemap index, which is not modified by any of the options
-    # passed to +group+.
+    # Returns a new LinkSet instance with the options passed in set on it.  All groups
+    # share the sitemap index, which is not affected by any of the options passed here.
     #
     # === Options
     # Any of the options to LinkSet.new.  Except for <tt>:public_path</tt> which is shared
@@ -127,9 +142,13 @@ module SitemapGenerator
     #
     # If you are not changing any of the location settings like <tt>filename<tt>,
     # <tt>sitemaps_path</tt>, <tt>sitemaps_host</tt> or <tt>sitemaps_namer</tt>
-    # the current sitemap will be used in the group.  All of the options you have
-    # specified which affect the way the links are generated will still be applied
-    # for the duration of the group.
+    # links you add within the group will be added to the current sitemap file (e.g. sitemap1.xml).
+    # If one of these options is specified, the current sitemap file is finalized
+    # and a new sitemap file started.
+    #
+    # Options like <tt>:default_host</tt> can be used and it will only affect the links
+    # within the group.  Links added outside of the group will revert to the previous
+    # +default_host+.
     def group(opts={}, &block)
       @created_group = true
       original_opts = opts.dup
@@ -230,6 +249,24 @@ module SitemapGenerator
       finalize_sitemap_index!
     end
 
+    # Return a boolean indicating hether to add a link to the sitemap index file
+    # to the current sitemap.  This points search engines to your Sitemap Index so
+    # they include it in the indexing of your site, but is not strictly neccessary.
+    # Default is `true`.  Turned off when `sitemaps_host` is set or within a `group()` block.
+    def include_index?
+      if default_host && sitemaps_host && sitemaps_host != default_host
+        false
+      else
+        @include_index
+      end
+    end
+
+    # Return a boolean indicating whether to automatically add the root url i.e. '/' to the
+    # current sitemap.  Default is `true`.  Turned off within a `group()` block.
+    def include_root?
+      !!@include_root
+    end
+
     protected
 
     # Set each option on this instance using accessor methods.  This will affect
@@ -273,8 +310,12 @@ module SitemapGenerator
     # Add default links if those options are turned on.  Record the fact that we have done so
     # in an instance variable.
     def add_default_links
-      sitemap.add('/', :lastmod => Time.now, :changefreq => 'always', :priority => 1.0, :host => @default_host) if include_root
-      sitemap.add(sitemap_index, :lastmod => Time.now, :changefreq => 'always', :priority => 1.0) if include_index
+      if include_root?
+        sitemap.add('/', :lastmod => Time.now, :changefreq => 'always', :priority => 1.0, :host => @default_host)
+      end
+      if include_index?
+        sitemap.add(sitemap_index, :lastmod => Time.now, :changefreq => 'always', :priority => 1.0)
+      end
       @added_default_links = true
     end
 
@@ -310,6 +351,15 @@ module SitemapGenerator
       @interpreter ||= SitemapGenerator::Interpreter.new(:link_set => self)
     end
 
+    # Reset this instance.  Keep the same options, but return to the same state
+    # as before an sitemaps were created.
+    def reset!
+      @sitemap_index = nil if @sitemap_index && @sitemap_index.finalized? && !@protect_index
+      @sitemap = nil if @sitemap && @sitemap.finalized?
+      self.sitemaps_namer.reset # start from 1
+      @added_default_links = false
+    end
+
     module LocationHelpers
       public
 
@@ -351,6 +401,9 @@ module SitemapGenerator
       # Set the host name, including protocol, that will be used on all links to your sitemap
       # files.  Useful when the server that hosts the sitemaps is not on the same host as
       # the links in the sitemap.
+      #
+      # Note that `include_index` will be turned off to avoid adding a link to a sitemap with
+      # a different host than the other links.
       def sitemaps_host=(value)
         @sitemaps_host = value
         update_location_info(:host, value)

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: sitemap_generator
 version: !ruby/object:Gem::Version 
-  hash: 11
+  hash: 9
   prerelease: 
   segments: 
   - 2
   - 1
-  - 0
-  version: 2.1.0
+  - 1
+  version: 2.1.1
 platform: ruby
 authors: 
 - Karl Varga
@@ -16,7 +16,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-08-31 00:00:00 -07:00
+date: 2011-09-19 00:00:00 -07:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency