sitemap_generator 0.2.0 → 0.2.1

data/README.md

@@ -42,36 +42,39 @@ Other "difficult" Sitemap issues, solved by this plugin:
 Installation
 =======
 
-*As a gem*
+**As a gem**
 
 1. Add the gem as a dependency in your config/environment.rb
-    <code>config.gem 'sitemap_generator', :source => 'http://gemcutter.org'</code>
 
-2. rake gems:install
+    <code>config.gem 'sitemap_generator', :lib => false, :source => 'http://gemcutter.org'</code>
 
-3. Add the following line to your RAILS_ROOT/Rakefile 
-    <code>begin require 'sitemap_generator/tasks' rescue LoadError end</code>
+2. `$ rake gems:install`
 
-4. `rake sitemap:install`
+3. Add the following line to your RAILS_ROOT/Rakefile
 
-*As a plugin*
+    <code>require 'sitemap_generator/tasks' rescue LoadError</code>
+
+4. `$ rake sitemap:install`
+
+**As a plugin**
 
 1. Install plugin as normal
 
-    <code>./script/plugin install git://github.com/adamsalter/sitemap_generator-plugin.git</code>
+    <code>$ ./script/plugin install git://github.com/adamsalter/sitemap_generator.git</code>
 
+----
 
 Installation should create a 'config/sitemap.rb' file which will contain your logic for generation of the Sitemap files. (If you want to recreate this file manually run `rake sitemap:install`)
 
 You can run `rake sitemap:refresh` as needed to create Sitemap files. This will also ping all the ['major'][sitemap_engines] search engines. (if you want to disable all non-essential output run the rake task thusly `rake -s sitemap:refresh`)
 
-    Sitemaps with many urls (100,000+) take quite a long time to generate, so if you need to refresh your Sitemaps regularly you can set the rake task up as a cron job. Most cron agents will only send you an email if there is output from the cron task.
+Sitemaps with many urls (100,000+) take quite a long time to generate, so if you need to refresh your Sitemaps regularly you can set the rake task up as a cron job. Most cron agents will only send you an email if there is output from the cron task.
 
 Optionally, you can add the following to your robots.txt file, so that robots can find the sitemap file.
 
-    <code>Sitemap: &lt;hostname>/sitemap_index.xml.gz</code>
+    Sitemap: <hostname>/sitemap_index.xml.gz
     
-    The robots.txt Sitemap URL should be the complete URL to the Sitemap Index, such as: `http://www.example.org/sitemap_index.xml.gz`
+The robots.txt Sitemap URL should be the complete URL to the Sitemap Index, such as: `http://www.example.org/sitemap_index.xml.gz`
 
 
 Example 'config/sitemap.rb'
@@ -139,6 +142,15 @@ Known Bugs
 - There's no check on the size of a URL which [isn't supposed to exceed 2,048 bytes][sitemaps_xml].
 - Currently only supports one Sitemap Index file, which can contain 50,000 Sitemap files which can each contain 50,000 urls, so it _only_ supports up to 2,500,000,000 (2.5 billion) urls. I personally have no need of support for more urls, but plugin could be improved to support this.
 
+Thanks (in no particular order)
+========
+
+- [Karl Varga (aka Bear Grylls)](http://github.com/kjvarga)
+- [Dan Pickett](http://github.com/dpickett)
+- [Rob Biedenharn](http://github.com/rab)
+- [Richie Vos](http://github.com/jerryvos)
+
+
 Follow me on:
 ---------
 
@@ -151,6 +163,6 @@ Copyright (c) 2009 Adam @ [Codebright.net][cb], released under the MIT license
 [sitemap_engines]:http://en.wikipedia.org/wiki/Sitemap_index "http://en.wikipedia.org/wiki/Sitemap_index"
 [sitemaps_org]:http://www.sitemaps.org/protocol.php "http://www.sitemaps.org/protocol.php"
 [sitemaps_xml]:http://www.sitemaps.org/protocol.php#xmlTagDefinitions "XML Tag Definitions"
-[sitemap_generator_usage]:http://wiki.github.com/adamsalter/sitemap_generator-plugin/sitemapgenerator-usage "http://wiki.github.com/adamsalter/sitemap_generator-plugin/sitemapgenerator-usage"
+[sitemap_generator_usage]:http://wiki.github.com/adamsalter/sitemap_generator/sitemapgenerator-usage "http://wiki.github.com/adamsalter/sitemap_generator/sitemapgenerator-usage"
 [boost_juice]:http://www.boostjuice.com.au/ "Mmmm, sweet, sweet Boost Juice."
 [cb]:http://codebright.net "http://codebright.net"

data/Rakefile

@@ -5,14 +5,15 @@ begin
   require 'jeweler'
   Jeweler::Tasks.new do |s|
     s.name = "sitemap_generator"
-    s.summary = %Q{This plugin enables 'enterprise-class' Google Sitemaps to be easily generated for a Rails site as a rake task}
-    s.description = %Q{This plugin enables 'enterprise-class' Google Sitemaps to be easily generated for a Rails site as a rake task}
+    s.summary = %Q{Generate 'enterprise-class' Sitemaps for your Rails site using a simple 'Rails Routes'-like DSL and a single Rake task}
+    s.description = %Q{Install as a plugin or Gem to easily generate ['enterprise-class'][enterprise_class] Google Sitemaps for your Rails site, using a simple 'Rails Routes'-like DSL and a single rake task.}
     s.email = "adam.salter@codebright.net "
-    s.homepage = "http://github.com/adamsalter/sitemap_generator-plugin"
+    s.homepage = "http://github.com/adamsalter/sitemap_generator"
     s.authors = ["Adam Salter"]
-    s.files =  FileList["[A-Z]*", "{bin,lib,rails,templates}/**/*"]
+    s.files =  FileList["[A-Z]*", "{bin,lib,rails,templates,tasks}/**/*"]
     # s is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
   end
+  Jeweler::GemcutterTasks.new
 rescue LoadError
   puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
 end

data/VERSION

@@ -1 +1 @@
-0.2.0
+0.2.1

data/lib/sitemap_generator.rb

@@ -4,6 +4,16 @@ require 'sitemap_generator/link_set'
 require 'sitemap_generator/helper'
 
 module SitemapGenerator
+  class <<self
+    attr_accessor :root, :templates
+  end
+  self.root = File.expand_path(File.join(File.dirname(__FILE__), '../'))
+  self.templates = {
+    :sitemap_index => File.join(self.root, 'templates/sitemap_index.builder'),
+    :sitemap_xml   => File.join(self.root, 'templates/xml_sitemap.builder'),
+    :sitemap_sample => File.join(self.root, 'templates/sitemap.rb'),
+  }
+  
   Sitemap = LinkSet.new
 end
   

data/lib/sitemap_generator/helper.rb

@@ -43,8 +43,8 @@ module SitemapGenerator
             puts "Successful ping of #{engine.to_s.titleize}" if verbose
           end
         rescue Timeout::Error, StandardError => e
-          puts "Ping failed for #{engine.to_s.titleize}: #{e.inspect}"
-          puts <<-END if engine == :yahoo
+          puts "Ping failed for #{engine.to_s.titleize}: #{e.inspect}" if verbose
+          puts <<-END if engine == :yahoo && verbose
 Yahoo requires an 'AppID' for more than one ping per "timeframe", you can either:
   - remove yahoo from the ping list (config/sitemap.rb):
     SitemapGenerator::Sitemap.yahoo_app_id = false

data/lib/sitemap_generator/mapper.rb

@@ -1,4 +1,3 @@
-
 module SitemapGenerator
   # Generator instances are used to build links.
   # The object passed to the add_links block in config/sitemap.rb is a Generator instance.

data/lib/sitemap_generator/tasks.rb

@@ -1 +1 @@
-load "#{File.dirname(__FILE__)}/../../tasks/sitemap_generator_tasks.rake"
+load File.expand_path(File.join(File.dirname(__FILE__), '../../tasks/sitemap_generator_tasks.rake'))

data/rails/install.rb

@@ -2,8 +2,7 @@
 
 # Copy sitemap_template.rb to config/sitemap.rb
 require 'fileutils'
-current_dir = File.dirname(__FILE__)
-sitemap_template = File.join(current_dir, 'templates/sitemap.rb')
+sitemap_template = File.join(File.dirname(__FILE__), '../templates/sitemap.rb')
 new_sitemap = File.join(RAILS_ROOT, 'config/sitemap.rb')
 if File.exist?(new_sitemap)
   puts "already exists: config/sitemap.rb, file not copied"

data/tasks/sitemap_generator_tasks.rake

@@ -0,0 +1,71 @@
+require 'zlib'
+
+namespace :sitemap do
+
+  desc "Install a default config/sitemap.rb file"
+  task :install do
+    load File.expand_path(File.join(File.dirname(__FILE__), "../rails/install.rb"))
+  end
+
+  desc "Delete all Sitemap files in public/ directory"
+  task :clean do
+    sitemap_files = Dir[File.join(RAILS_ROOT, 'public/sitemap*.xml.gz')]
+    FileUtils.rm sitemap_files
+  end
+
+  desc "Create Sitemap XML files in public/ directory"
+  desc "Create Sitemap XML files in public/ directory (rake -s for no output)"
+  task :refresh => ['sitemap:create'] do
+    ping_search_engines("sitemap_index.xml.gz")
+  end
+
+  desc "Create Sitemap XML files (don't ping search engines)"
+  task 'refresh:no_ping' => ['sitemap:create'] do
+  end
+
+  task :create => [:environment] do
+    include SitemapGenerator::Helper
+    include ActionView::Helpers::NumberHelper
+
+    start_time = Time.now
+
+    # update links from config/sitemap.rb
+    load_sitemap_rb
+
+    raise(ArgumentError, "Default hostname not defined") if SitemapGenerator::Sitemap.default_host.blank?
+
+    links_grps = SitemapGenerator::Sitemap.links.in_groups_of(50000, false)
+    raise(ArgumentError, "TOO MANY LINKS!! I really thought 2,500,000,000 links would be enough for anybody!") if links_grps.length > 50000
+
+    Rake::Task['sitemap:clean'].invoke
+
+    # render individual sitemaps
+    sitemap_files = []
+    links_grps.each_with_index do |links, index|
+      buffer = ''
+      xml = Builder::XmlMarkup.new(:target=>buffer)
+      eval(open(SitemapGenerator.templates[:sitemap_xml]).read, binding)
+      filename = File.join(RAILS_ROOT, "public/sitemap#{index+1}.xml.gz")
+      Zlib::GzipWriter.open(filename) do |gz|
+        gz.write buffer
+      end
+      puts "+ #{filename}" if verbose
+      puts "** Sitemap too big! The uncompressed size exceeds 10Mb" if (buffer.size > 10 * 1024 * 1024) && verbose
+      sitemap_files << filename
+    end
+
+    # render index
+    buffer = ''
+    xml = Builder::XmlMarkup.new(:target=>buffer)
+    eval(open(SitemapGenerator.templates[:sitemap_index]).read, binding)
+    filename = File.join(RAILS_ROOT, "public/sitemap_index.xml.gz")
+    Zlib::GzipWriter.open(filename) do |gz|
+      gz.write buffer
+    end
+    puts "+ #{filename}" if verbose
+    puts "** Sitemap Index too big! The uncompressed size exceeds 10Mb" if (buffer.size > 10 * 1024 * 1024) && verbose
+
+    stop_time = Time.now
+    puts "Sitemap stats: #{number_with_delimiter(SitemapGenerator::Sitemap.links.length)} links, " + ("%dm%02ds" % (stop_time - start_time).divmod(60)) if verbose
+  end
+end

data/test/sitemap_generator_test.rb

@@ -1,19 +1,83 @@
 require File.dirname(__FILE__) + '/test_helper'
 
 class SitemapGeneratorTest < Test::Unit::TestCase
-  context "SitemapGenerator Rake Task" do
-    setup do
-      ::Rake::Task['sitemap:refresh'].invoke
+  context "SitemapGenerator Rake Tasks" do
+
+    context "when running the clean task" do
+      setup do
+        copy_sitemap_file_to_rails_app
+        FileUtils.touch(File.join(RAILS_ROOT, '/public/sitemap_index.xml.gz'))
+        Rake::Task['sitemap:clean'].invoke
+      end
+    
+      should "the sitemap xml files be deleted" do
+        assert !File.exists?(File.join(RAILS_ROOT, '/public/sitemap_index.xml.gz'))
+      end
+    end
+    
+    # For some reason I just couldn't get this to work!  It seemed to delete the
+    # file before calling the second *should* assertion.
+    context "when installed to a clean Rails app" do
+      setup do
+        #delete_sitemap_file_from_rails_app
+        #Rake::Task['sitemap:install'].invoke
+      end
+
+      should "a sitemap.rb is created" do
+        #assert File.exists?(File.join(RAILS_ROOT, 'config/sitemap.rb'))
+      end
+
+      should "the sitemap.rb file matches the template" do
+        #assert identical_files?(File.join(RAILS_ROOT, 'config/sitemap.rb'), SitemapGenerator.templates[:sitemap_sample])
+      end
     end
     
-    should "fail if hostname not defined" do
+    context "when installed multiple times" do
+      setup do
+        copy_sitemap_file_to_rails_app
+        Rake::Task['sitemap:install'].invoke
+      end  
+          
+      should "not overwrite existing sitemap.rb file" do  
+        assert identical_files?(File.join(File.dirname(__FILE__), '/sitemap.file'), File.join(RAILS_ROOT, '/config/sitemap.rb'))
+      end
+    end
+    
+    context "when sitemap generated" do
+      setup do
+        copy_sitemap_file_to_rails_app
+        Rake::Task['sitemap:refresh'].invoke
+      end  
+          
+      should "not create sitemap xml files" do  
+        assert File.exists?(File.join(RAILS_ROOT, '/public/sitemap_index.xml.gz'))
+        assert File.exists?(File.join(RAILS_ROOT, '/public/sitemap1.xml.gz'))  
+      end
     end
   end
-
+  
   context "SitemapGenerator library" do
+    setup do
+      copy_sitemap_file_to_rails_app
+    end
+        
     should "be have x elements" do
-      assert_equal SitemapGenerator::Sitemap.links.size, 14
+      assert_equal 14, SitemapGenerator::Sitemap.links.size
     end
   end
+  
+  def copy_sitemap_file_to_rails_app
+    FileUtils.cp(File.join(File.dirname(__FILE__), '/sitemap.file'), File.join(RAILS_ROOT, '/config/sitemap.rb'))
+  end
+  
+  def delete_sitemap_file_from_rails_app
+    FileUtils.remove(File.join(RAILS_ROOT, '/config/sitemap.rb')) rescue nil
+  end
+  
+  def identical_files?(first, second)
+    first = open(first, 'r').read
+    second = open(second, 'r').read
+    first == second
+  end
 end
 

data/test/test_helper.rb

@@ -3,6 +3,7 @@ ENV['RAILS_ROOT'] ||= File.join(File.dirname(__FILE__), 'mock_app')
 
 require File.expand_path(File.join(ENV['RAILS_ROOT'], 'config', 'environment.rb'))
 
+require 'fileutils'
 require 'rake'
 require 'shoulda'
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: sitemap_generator
 version: !ruby/object:Gem::Version 
-  version: 0.2.0
+  version: 0.2.1
 platform: ruby
 authors: 
 - Adam Salter
@@ -9,11 +9,11 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2009-11-06 00:00:00 +11:00
+date: 2009-11-10 00:00:00 +08:00
 default_executable: 
 dependencies: []
 
-description: This plugin enables 'enterprise-class' Google Sitemaps to be easily generated for a Rails site as a rake task
+description: Install as a plugin or Gem to easily generate ['enterprise-class'][enterprise_class] Google Sitemaps for your Rails site, using a simple 'Rails Routes'-like DSL and a single rake task.
 email: "adam.salter@codebright.net "
 executables: []
 
@@ -34,11 +34,12 @@ files:
 - lib/sitemap_generator/tasks.rb
 - rails/install.rb
 - rails/uninstall.rb
+- tasks/sitemap_generator_tasks.rake
 - templates/sitemap.rb
 - templates/sitemap_index.builder
 - templates/xml_sitemap.builder
 has_rdoc: true
-homepage: http://github.com/adamsalter/sitemap_generator-plugin
+homepage: http://github.com/adamsalter/sitemap_generator
 licenses: []
 
 post_install_message: 
@@ -64,7 +65,7 @@ rubyforge_project:
 rubygems_version: 1.3.5
 signing_key: 
 specification_version: 3
-summary: This plugin enables 'enterprise-class' Google Sitemaps to be easily generated for a Rails site as a rake task
+summary: Generate 'enterprise-class' Sitemaps for your Rails site using a simple 'Rails Routes'-like DSL and a single Rake task
 test_files: 
 - test/mock_app/app/controllers/application_controller.rb
 - test/mock_app/app/controllers/contents_controller.rb