bcms_sitemap 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/LICENSE.txt +674 -0
- data/README.markdown +167 -0
- data/app/controllers/cms/search_engines_controller.rb +19 -0
- data/app/controllers/sitemaps_controller.rb +22 -0
- data/app/helpers/sitemaps_helper.rb +13 -0
- data/app/models/search_engine.rb +69 -0
- data/app/views/cms/search_engines/_form.html.erb +17 -0
- data/app/views/cms/search_engines/edit.html.erb +3 -0
- data/app/views/cms/search_engines/index.html.erb +51 -0
- data/app/views/cms/search_engines/new.html.erb +3 -0
- data/app/views/cms/shared/_admin_sidebar.html.erb +31 -0
- data/app/views/sitemaps/index.builder +10 -0
- data/app/views/sitemaps/model.builder +11 -0
- data/app/views/sitemaps/news_articles.builder +11 -0
- data/config/initializers/bcms_sitemap.rb +4 -0
- data/config/initializers/init_module.rb +1 -0
- data/db/migrate/20100212083005_create_search_engines.rb +24 -0
- data/lib/bcms_sitemap.rb +3 -0
- data/lib/bcms_sitemap/routes.rb +10 -0
- data/lib/bcms_sitemap/sitemap_submitter.rb +110 -0
- data/lib/tasks/bcms_sitemap.rake +4 -0
- data/lib/tasks/sitemap.rake +34 -0
- data/rails/init.rb +6 -0
- data/test/functional/sitemaps_controller_test.rb +139 -0
- data/test/integration/sitemap_submitter_test.rb +85 -0
- data/test/performance/browsing_test.rb +9 -0
- data/test/test_factory.rb +29 -0
- data/test/test_helper.rb +39 -0
- data/test/unit/helpers/sitemaps_helper_test.rb +4 -0
- data/test/unit/search_engine_test.rb +126 -0
- metadata +93 -0
data/README.markdown
ADDED
@@ -0,0 +1,167 @@
|
|
1
|
+
**WARNING: This module has not yet been tested in a live production environment**
|
2
|
+
|
3
|
+
**NOTE: Gem has not been built yet.**
|
4
|
+
|
5
|
+
# BCMS\_sitemap BrowserCMS sitemap submission module
|
6
|
+
|
7
|
+
This is a general purpose sitemap submission tool for BrowserCMS. It will automatically submit your site's sitemap to the search engines of
|
8
|
+
your preference. Once setup properly, it will submit the following url to your search engine: http://..yoursite../sitemaps.xml. This document
|
9
|
+
contains information about when your models were changed last time and urls for fetching your models sitemaps, e.g. the search engine will be
|
10
|
+
instructed to fetch http://..yoursite../sitemaps/pages.xml for your pages model.
|
11
|
+
|
12
|
+
Search engines wants sitemaps submitted no more often than once an hour. You will need to setup a cron or a backgroundrb job to submit your
|
13
|
+
sitemaps.
|
14
|
+
|
15
|
+
If you deploy remotely with capistrano you will want to change your deployment routines so that signatory (verification) files are
|
16
|
+
re-created after deployment.
|
17
|
+
|
18
|
+
**Limitations:** Search engines do not accept more than 50,000 page references in a sitemap file. If you expect to have more than 50,000
|
19
|
+
active records in your models, then, you will need to implement paging in the module.
|
20
|
+
|
21
|
+
|
22
|
+
## Installation instructions
|
23
|
+
|
24
|
+
**Install the gem**
|
25
|
+
|
26
|
+
sudo gem install bcms_sitemap
|
27
|
+
|
28
|
+
**Add it to your project**
|
29
|
+
|
30
|
+
Edit the config/environment.rb file and add the following after config.gem 'browsercms':
|
31
|
+
|
32
|
+
config.gem 'bcms_sitemap'
|
33
|
+
|
34
|
+
**Install the new module**
|
35
|
+
|
36
|
+
From the root directory of your app, run the following:
|
37
|
+
|
38
|
+
$ script/generate browser_cms
|
39
|
+
|
40
|
+
This will add the necessary files to you application. In particular, you may want to change the settings in
|
41
|
+
the file config/initializers/bcms\_sitemap.rb. (See also customization below)
|
42
|
+
|
43
|
+
Edit routes.rb file - add the following *before* map.routes\_for\_browser\_cms:
|
44
|
+
|
45
|
+
map.routes_for_bcms_sitemap
|
46
|
+
|
47
|
+
**Run the migrations**
|
48
|
+
|
49
|
+
$ rake db:migrate
|
50
|
+
|
51
|
+
**Register sitemap in robots.txt (optional)**
|
52
|
+
|
53
|
+
Add the following line to your robots.txt
|
54
|
+
|
55
|
+
Sitemap: http://...yoursite.../sitemaps.xml
|
56
|
+
|
57
|
+
**Additional setup**
|
58
|
+
|
59
|
+
In order for your sitemap to be submitted
|
60
|
+
|
61
|
+
* you will need to register your the search engine you would like to submit to
|
62
|
+
* You will need to setup a job that submits your sitemap.
|
63
|
+
|
64
|
+
See next section - configuration.
|
65
|
+
|
66
|
+
|
67
|
+
## Configuration
|
68
|
+
|
69
|
+
### Registering your search engines
|
70
|
+
|
71
|
+
To submit sitemap to search engines, you will need to enable the search engines in the search engines administrative pages. Some search
|
72
|
+
engines requires that you authenticate with the site. Where authentication is required, such as for Google, Yahoo, and Bing, you will have to
|
73
|
+
register some kind of verification, normally specified, using a verification file.
|
74
|
+
|
75
|
+
The search engine section in the admin pages will allow you to enter the required name and content for the verification files,
|
76
|
+
and automatically create these upon saving the record for the individual search engines.
|
77
|
+
|
78
|
+
For more information see the individual sites:
|
79
|
+
|
80
|
+
[Ask](http://about.ask.com/en/docs/about/webmasters.shtml),
|
81
|
+
[Google](http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156184),
|
82
|
+
[Yahoo](http://help.yahoo.com/l/us/yahoo/smallbusiness/store/promote/sitemap/sitemap-16.html) and
|
83
|
+
[yahoo submissions](https://siteexplorer.search.yahoo.com/submit), [Live/Bing](http://www.bing.com/webmaster)
|
84
|
+
|
85
|
+
[Moreover](http://moreover.com) does not require authentication. (Not really a search engine, but provides news many to companies)
|
86
|
+
|
87
|
+
[See also](http://www.hariesdesign.com/tips-and-trick/6-seo/46-submit-a-sitemap-to-top-5-search-engine) http://www.hariesdesign.com/tips-and-trick/6-seo/46-submit-a-sitemap-to-top-5-search-engine
|
88
|
+
|
89
|
+
### Setting up your submission job
|
90
|
+
You can use either cron or backgroundrb to submit your sitemap. My preference is cron.
|
91
|
+
On a Ubuntu server you would do the following:
|
92
|
+
|
93
|
+
sudo vi /etc/cron.hourly/submit_sitemaps
|
94
|
+
|
95
|
+
Add the following:
|
96
|
+
|
97
|
+
#!/usr/bin/env ruby
|
98
|
+
require 'fileutils'
|
99
|
+
|
100
|
+
APP_ROOT="/...yoursite..."
|
101
|
+
RAILS_ENV="production"
|
102
|
+
RAKE_OPTS='-q -s'
|
103
|
+
Dir.chdir "#{APP_ROOT}/current"
|
104
|
+
`RAILS_ENV=#{RAILS_ENV} rake #{RAKE_OPTS} sitemap:submit`
|
105
|
+
|
106
|
+
Make it executable:
|
107
|
+
|
108
|
+
sudo chmod 755 /etc/cron.hourly/submit_sitemaps
|
109
|
+
|
110
|
+
## Customizations (adding new models)
|
111
|
+
|
112
|
+
The sitemap submission facility has been setup with **pages** and **article\_news**. If you have more models that you want to add
|
113
|
+
or remove, you need to change the bcms_sitemap.rb file in your config/initializers directory:
|
114
|
+
|
115
|
+
1. State what models to publish as a hash.
|
116
|
+
2. The keys are plural name of the model.
|
117
|
+
3. The values should be the scope to be used, formed as string
|
118
|
+
|
119
|
+
Cms::SitemapSubmitter.publish_models = {:pages => 'published.not_hidden' :news_articles => 'released'}
|
120
|
+
|
121
|
+
The bcms\_sitemap module will backport your model to access the scopes you define.
|
122
|
+
It will also automatically respond to a defined route for your model, e.g. if you add 'feeds', it will publish the feeds sitemap at
|
123
|
+
http://...yoursite.../sitemaps/feeds.xml
|
124
|
+
|
125
|
+
By default the view is generated with the view app/views/sitemaps/models.builder (located with the gem). If you want differenciated
|
126
|
+
priorities or change modification frequencies, you can create your own builder. If so, you will have to give it a corresponding name, e.g. for
|
127
|
+
feeds you would name the view feeds.builder. Below is the source code for the model.builder.
|
128
|
+
|
129
|
+
xml.instruct! :xml, :version => "1.0", :encoding => "UTF-8"
|
130
|
+
xml.urlset :xmlns => "http://www.sitemaps.org/schemas/sitemap/0.9" do
|
131
|
+
@objects.each do |object|
|
132
|
+
xml.url do
|
133
|
+
xml.loc xml_url(object)
|
134
|
+
xml.lastmod xml_lastmod(object)
|
135
|
+
xml.changefreq 'weekly'
|
136
|
+
xml.priority '0.5'
|
137
|
+
end
|
138
|
+
end
|
139
|
+
end
|
140
|
+
|
141
|
+
The controller instanciates a variable with the model name pluralized, if you would prefere to use that, i.e. if the builder you are
|
142
|
+
creating is for feeds, you may use @feeds instead of @objects.
|
143
|
+
|
144
|
+
## Rake tasks
|
145
|
+
|
146
|
+
There are three rake tasks:
|
147
|
+
|
148
|
+
* **sitemap:submit** which submits the sitemaps
|
149
|
+
* **sitemap:status** which lists the status of submitted sitemaps to the search engines
|
150
|
+
* **sitemap:verify_signatories** which re-creates signatory (verification) files after deployment, if necessary.
|
151
|
+
|
152
|
+
If you deploy using capistrano, you may want to add the sitemap:verify_signatories task deployment.rb
|
153
|
+
|
154
|
+
namespace :sitemap, :roles => :app do
|
155
|
+
task :verify_signatories do
|
156
|
+
rake = fetch(:rake, "rake")
|
157
|
+
rails_env = fetch(:rails_env, "production")
|
158
|
+
run "cd \#{current_path}; \#{rake} RAILS_ENV=\#{rails_env} sitemap:verify_signatories"
|
159
|
+
end
|
160
|
+
end
|
161
|
+
after "deploy:finalize_update", "sitemap:verify_signatories"
|
162
|
+
|
163
|
+
|
164
|
+
# Considerations for the future
|
165
|
+
|
166
|
+
Currently the module is placing a template for rendering the search engines in the administration interface, due to inflexible module
|
167
|
+
facilities in BrowserCMS. (Should be on the backlog for the browser cms team). Meanwhile, that view has to be stored locally.
|
@@ -0,0 +1,19 @@
|
|
1
|
+
class Cms::SearchEnginesController < Cms::ResourceController
|
2
|
+
layout 'cms/administration'
|
3
|
+
check_permissions :administrate
|
4
|
+
before_filter :set_menu_section
|
5
|
+
protected
|
6
|
+
def show_url
|
7
|
+
index_url
|
8
|
+
end
|
9
|
+
|
10
|
+
def order_by_column
|
11
|
+
"name"
|
12
|
+
end
|
13
|
+
|
14
|
+
private
|
15
|
+
def set_menu_section
|
16
|
+
@menu_section = 'search_engines'
|
17
|
+
end
|
18
|
+
|
19
|
+
end
|
@@ -0,0 +1,22 @@
|
|
1
|
+
class SitemapsController < ApplicationController
|
2
|
+
caches_page :index, :model
|
3
|
+
|
4
|
+
def index
|
5
|
+
@entries = {}
|
6
|
+
Cms::SitemapSubmitter.models.each do |model|
|
7
|
+
last_update = model.classify.constantize.bcms_sitemap_last_update
|
8
|
+
@entries[model.pluralize] = last_update if last_update
|
9
|
+
end
|
10
|
+
end
|
11
|
+
|
12
|
+
def model
|
13
|
+
model = params[:model]
|
14
|
+
@objects = model.classify.constantize.bcms_sitemap_scope
|
15
|
+
instance_variable_set("@#{model}", @objects) # Template usually wants @pages, @news_articles etc
|
16
|
+
if Rails.root.join('app','views','sitemaps',"#{model}.builder").exist?
|
17
|
+
render "#{model}.builder"
|
18
|
+
else
|
19
|
+
render 'model.builder'
|
20
|
+
end
|
21
|
+
end
|
22
|
+
end
|
@@ -0,0 +1,69 @@
|
|
1
|
+
class SearchEngine < ActiveRecord::Base
|
2
|
+
validates_presence_of :name
|
3
|
+
validates_uniqueness_of :name, :case_sensitive => false
|
4
|
+
validates_presence_of :url, :if => :enabled?
|
5
|
+
|
6
|
+
after_save :manage_signatory_file
|
7
|
+
after_destroy :remove_signatory_file
|
8
|
+
|
9
|
+
named_scope :enabled, :conditions => ['enabled = ?', true]
|
10
|
+
|
11
|
+
class << self
|
12
|
+
def last_submit
|
13
|
+
minimum(:submitted_at)
|
14
|
+
end
|
15
|
+
|
16
|
+
def verify_signatories!
|
17
|
+
all.each do |search_engine|
|
18
|
+
search_engine.send(:create_signatory_file) unless search_engine.has_signatory_file?
|
19
|
+
end
|
20
|
+
end
|
21
|
+
|
22
|
+
def signatory_folder
|
23
|
+
File.join(Rails.root, 'public')
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
def submit
|
28
|
+
if status = Cms::SitemapSubmitter.submit(self)
|
29
|
+
self.update_attributes(:submitted_at => Time.now, :last_status => status)
|
30
|
+
end
|
31
|
+
end
|
32
|
+
|
33
|
+
def signatory_folder
|
34
|
+
self.class.signatory_folder
|
35
|
+
end
|
36
|
+
|
37
|
+
def signatory_file_name
|
38
|
+
verification_file && !verification_file.empty? && File.join(signatory_folder, verification_file )
|
39
|
+
end
|
40
|
+
|
41
|
+
def has_signatory_file?
|
42
|
+
signatory_file_name && File.exists?(signatory_file_name)
|
43
|
+
end
|
44
|
+
|
45
|
+
protected
|
46
|
+
def manage_signatory_file
|
47
|
+
if verification_file_changed? || verification_content_changed?
|
48
|
+
unless verification_file_was.nil?
|
49
|
+
# new file_name - just remove the old one
|
50
|
+
File.delete File.join(signatory_folder, verification_file_was )
|
51
|
+
end
|
52
|
+
create_signatory_file
|
53
|
+
end
|
54
|
+
end
|
55
|
+
|
56
|
+
def create_signatory_file
|
57
|
+
if verification_file
|
58
|
+
File.atomic_write(signatory_file_name) do |file|
|
59
|
+
file.write(verification_content || '')
|
60
|
+
end
|
61
|
+
end
|
62
|
+
end
|
63
|
+
|
64
|
+
def remove_signatory_file
|
65
|
+
if signatory_file_name
|
66
|
+
File.delete signatory_file_name
|
67
|
+
end
|
68
|
+
end
|
69
|
+
end
|
@@ -0,0 +1,17 @@
|
|
1
|
+
<%= content_for :html_head, stylesheet_link_tag('cms/form_layout') %>
|
2
|
+
<% form_for([:cms, @search_engine]) do |f| %>
|
3
|
+
<%= f.error_messages %>
|
4
|
+
<%= f.cms_text_field :name %>
|
5
|
+
<div class="fields checkbox_group">
|
6
|
+
<%= f.label :enabled, 'Enabled?' %>
|
7
|
+
<%= f.check_box :enabled %>
|
8
|
+
</div>
|
9
|
+
<%= f.cms_text_field :url, :label => 'Submission url',
|
10
|
+
:instructions => 'This should be the complete submission url, ending with "=". The local sitemap url will be added to the end of this url' %>
|
11
|
+
<%= f.cms_text_field :verification_file,
|
12
|
+
:instructions => 'Input filename. The system assumes that the file should reside in the public folder. Upon save, the file will be stored in that location with thw content defined below.' %>
|
13
|
+
<%= f.cms_text_area :verification_content, :label => "Verification file content" %>
|
14
|
+
<div class="buttons">
|
15
|
+
<%= lt_button_wrapper(f.submit("Save", :class => "submit")) %>
|
16
|
+
</div>
|
17
|
+
<% end %>
|
@@ -0,0 +1,51 @@
|
|
1
|
+
<% @page_title = @toolbar_title = "List Search Engines" %>
|
2
|
+
<% content_for(:html_head) do %>
|
3
|
+
<% javascript_tag do %>
|
4
|
+
jQuery(function($){
|
5
|
+
$('table.data tbody tr').hover(function(){
|
6
|
+
$(this).addClass('hover')
|
7
|
+
}, function(){
|
8
|
+
$(this).removeClass('hover')
|
9
|
+
}).click(function(){
|
10
|
+
var match = this.id.match(/(.*)_(\d+)/)
|
11
|
+
var type = match[1];
|
12
|
+
var id = match[2];
|
13
|
+
$('table.data tbody tr').removeClass('selected');
|
14
|
+
$(this).addClass('selected');
|
15
|
+
$('#edit_button').removeClass('disabled').attr('href', '/cms/search_engines/'+id+'/edit');
|
16
|
+
$('#delete_button').removeClass('disabled').attr('href', '/cms/search_engines/'+id)
|
17
|
+
})
|
18
|
+
})
|
19
|
+
<% end %>
|
20
|
+
<% end %>
|
21
|
+
<% content_for :functions do %>
|
22
|
+
<%= link_to(span_tag(" Add "), new_cms_search_engine_path, :class => "button") %>
|
23
|
+
<%= link_to(span_tag(" Edit "), '#', :id => 'edit_button', :class => "button disabled") %>
|
24
|
+
<%= link_to span_tag("<span class=\"delete_img\"> </span>Delete"), "#", :id => "delete_button", :class => "button disabled http_delete", :onclick => "return confirm('Are you sure you want to delete this search engine?')" %>
|
25
|
+
<% end %>
|
26
|
+
<div class="roundedcorners">
|
27
|
+
<table class="data">
|
28
|
+
<thead>
|
29
|
+
<tr>
|
30
|
+
<th class="first"></th>
|
31
|
+
<th><div class="dividers">Name</div></th>
|
32
|
+
<th><div class="dividers">Enabled</div></th>
|
33
|
+
<th><div>Submitted At</div></th>
|
34
|
+
<th class="last"></th>
|
35
|
+
</tr>
|
36
|
+
</thead>
|
37
|
+
<% @search_engines.each do |search_engine| %>
|
38
|
+
<tr id="search_engine_<%= search_engine.id %>">
|
39
|
+
<td class="first"></td>
|
40
|
+
<td><div class="dividers"><%=h search_engine.name %></div></td>
|
41
|
+
<td><div class="dividers"><%= search_engine.enabled? ? 'Enabled' : '' %></div></td>
|
42
|
+
<td><div><%= search_engine.submitted_at && search_engine.submitted_at.to_s(:date) %></div></td>
|
43
|
+
<td class="last"></td>
|
44
|
+
</tr>
|
45
|
+
<% end %>
|
46
|
+
</table>
|
47
|
+
<div class="tl"></div>
|
48
|
+
<div class="tr"></div>
|
49
|
+
<div class="bl"></div>
|
50
|
+
<div class="br"></div>
|
51
|
+
</div>
|
@@ -0,0 +1,31 @@
|
|
1
|
+
<div class="first">
|
2
|
+
<h3>Users</h3>
|
3
|
+
<ul>
|
4
|
+
<li<%= ' class="open"' if @menu_section == 'users' %>><%= link_to "Users", cms_users_path %></li>
|
5
|
+
<li<%= ' class="open"' if @menu_section == 'groups' %>><%= link_to "Groups", cms_groups_path %></li>
|
6
|
+
</ul>
|
7
|
+
</div>
|
8
|
+
<div class="bottom_cap bottom_pad"></div>
|
9
|
+
|
10
|
+
<div class="top_cap"></div>
|
11
|
+
<div>
|
12
|
+
<h3>Templates</h3>
|
13
|
+
<ul>
|
14
|
+
<li<%= ' class="open"' if @menu_section == 'page_partials' %>><%= link_to "Page Partials", cms_page_partials_path %></li>
|
15
|
+
<li<%= ' class="open"' if @menu_section == 'page_templates' %>><%= link_to "Page Templates", cms_page_templates_path %></li>
|
16
|
+
<li<%= ' class="open"' if @menu_section == 'page_routes' %>><%= link_to "Page Routes", cms_page_routes_path %></li>
|
17
|
+
</ul>
|
18
|
+
</div>
|
19
|
+
<div class="bottom_cap bottom_pad"></div>
|
20
|
+
|
21
|
+
<div class="top_cap"></div>
|
22
|
+
<div>
|
23
|
+
<h3>Tools</h3>
|
24
|
+
<ul>
|
25
|
+
<li<%= ' class="open"' if @menu_section == 'caching' %>><%= link_to "Page Caching", cms_cache_path %></li>
|
26
|
+
<li<%= ' class="open"' if @menu_section == 'redirects' %>><%= link_to "Redirects", cms_redirects_path %></li>
|
27
|
+
<li<%= ' class="open"' if @menu_section == 'email_messages' %>><%= link_to "Email Messages", cms_email_messages_path %></li>
|
28
|
+
<li<%= ' class="open"' if @menu_section == 'search_engines' %>><%= link_to "Search Engine Submission", cms_search_engines_path %></li>
|
29
|
+
</ul>
|
30
|
+
</div>
|
31
|
+
<div class="bottom_cap"></div>
|
@@ -0,0 +1,10 @@
|
|
1
|
+
xml.instruct! :xml, :version => "1.0", :encoding => "UTF-8"
|
2
|
+
xml.sitemapindex :xmlns => "http://www.sitemaps.org/schemas/sitemap/0.9" do
|
3
|
+
@entries.sort { |a,b| a[0] <=> b[0] }.each do |model,date|
|
4
|
+
xml.sitemap do
|
5
|
+
# xml.loc "http://#{SITE_DOMAIN}/sitemaps/#{path}.xml"
|
6
|
+
xml.loc "#{sitemap_for_model_url(model)}"
|
7
|
+
xml.lastmod date.xmlschema
|
8
|
+
end
|
9
|
+
end
|
10
|
+
end
|
@@ -0,0 +1,11 @@
|
|
1
|
+
xml.instruct! :xml, :version => "1.0", :encoding => "UTF-8"
|
2
|
+
xml.urlset :xmlns => "http://www.sitemaps.org/schemas/sitemap/0.9" do
|
3
|
+
@objects.each do |object|
|
4
|
+
xml.url do
|
5
|
+
xml.loc xml_url(object)
|
6
|
+
xml.lastmod xml_lastmod(object)
|
7
|
+
xml.changefreq 'weekly'
|
8
|
+
xml.priority '0.5'
|
9
|
+
end
|
10
|
+
end
|
11
|
+
end
|
@@ -0,0 +1,11 @@
|
|
1
|
+
xml.instruct! :xml, :version => "1.0", :encoding => "UTF-8"
|
2
|
+
xml.urlset :xmlns => "http://www.sitemaps.org/schemas/sitemap/0.9" do
|
3
|
+
@articles.each do |object|
|
4
|
+
xml.url do
|
5
|
+
xml.loc news_article_url(object.route_params)
|
6
|
+
xml.lastmod xml_lastmod(object)
|
7
|
+
xml.changefreq object.archived? ? 'never' : 'weekly'
|
8
|
+
xml.priority object.archived? ? 0.4 : 0.7
|
9
|
+
end
|
10
|
+
end
|
11
|
+
end
|