adsense_crawler_for_private 1.0.0

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright 2011 YOURNAME
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,112 @@
+# AdsenseCrawlerForPrivate
+
+Easy way to enable adsense crawler to login and see private or custom pages. Basically one custom login filter.
+
+[<img src="https://secure.travis-ci.org/holli/adsense_crawler_for_private.png" />](http://travis-ci.org/holli/adsense_crawler_for_private)
+
+## Usage
+
+
+**Initialize:** In RAILS_ROOT/config/initializers/adsense_crawler_for_private.rb
+
+```
+# Configure adsense_crawler_for_private
+
+AdsenseCrawlerForPrivate.cookie_name = "adsense_crawler"
+AdsenseCrawlerForPrivate.cookie_domain = :all
+AdsenseCrawlerForPrivate.crawler_name = "test_name"
+AdsenseCrawlerForPrivate.crawler_password = "test_password"
+
+# If you are paranoid you can specify ip addresses that are ok to the crawlers to access
+# AdsenseCrawlerForPrivate.ip_ranges = [IPAddr.new("127.0.0.1"), IPAddr.new("192.168.0.1/20")]
+
+```
+
+**Routes:**
+
+```
+  # Give url "http://domain.dom/adsense_crawler_for_private/login" for crawlers to log in
+  # or directly "http://domain.dom/adsense_crawler_for_private/login?name=test_name&password=test_password"
+  mount AdsenseCrawlerForPrivate::Engine => "/adsense_crawler_for_private"
+```
+
+**Rendering etc usage:** in controller define what to render for crawlers
+
+```
+
+def SomeController << ApplicationController
+
+  before_filter :adsense_crawler_private_specific_page # normal authentication filters after this one
+
+  def adsense_crawler_private_specific_page
+    if AdsenseCrawlerForPrivate.login_check(cookies, request)
+      # here info how to render page for crawler
+      # e.g render 'crawler_ad_page'
+      # or creating a dummy login info
+      return false # so that normal authentication filters etc are not effective
+    end
+  end
+
+  # Here would be rest of the controller
+
+end
+
+```
+
+Or if you want only to check alongside other authentication you can call in your
+own authentication filters AdsenseCrawlerForPrivate.login_check(request)-method
+directly.
+
+
+### Testing your own stuff
+
+You can test your filters by setting cookie in the same way as in AdsenseCrawlerLoginController#login .
+
+If you have enabled ip_ranges option you have to make sure that the ip that tests use is enabled for crawlers.
+
+```
+test "here would be a test for logged crawler in functional tests"
+  # Dummy login for crawler, These should be configured in initializers
+  crawler_name="adsense_crawler"; crawler_password = "adsense_pass";
+  @request.remote_addr = ip = "127.0.0.1"
+
+  # In some frameworks cookies.signed would be enough. Some will need you to sign the cookie by yourself.
+  # If you have a better way, let me know
+  # same as cookies.signed[AdsenseCrawlerForPrivate.cookie_name] = AdsenseCrawlerForPrivate.cookie_hash(crawler_name, ip)
+  @request.cookies[AdsenseCrawlerForPrivate.cookie_name] =
+        ActiveSupport::MessageVerifier.new(Dummy::Application.config.secret_token).generate(
+            AdsenseCrawlerForPrivate.cookie_str(crawler_name, crawler_password, 2.days.from_now, ip))
+
+
+  #Normal test in here
+  get :index
+  assert_response :success
+  assert response.body.include?("Hi normal crawler")
+end
+
+test "here would be a test for non-logged crawler"
+  get :index
+  assert_response :success
+  assert response.body.include?("Hi normal user")
+end
+```
+
+### robots.txt
+
+Remember to update robots.txt if you have previously forbidden adsense to crawl certain pages.
+
+## Requirements
+
+Gem has been tested with ruby 1.8.7, 1.9.2 and Rails 3.1.
+
+[<img src="https://secure.travis-ci.org/holli/adsense_crawler_for_private.png" />](http://travis-ci.org/holli/adsense_crawler_for_private)
+
+http://travis-ci.org/#!/holli/adsense_crawler_for_private
+
+## Support
+
+Submit suggestions or feature requests as a GitHub Issue or Pull Request. Remember to update tests. Tests are quite extensive.
+
+## License
+
+Released under the MIT license (http://www.opensource.org/licenses/mit-license.php)

data/Rakefile

@@ -0,0 +1,39 @@
+#!/usr/bin/env rake
+begin
+  require 'bundler/setup'
+rescue LoadError
+  puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
+end
+begin
+  require 'rdoc/task'
+rescue LoadError
+  require 'rdoc/rdoc'
+  require 'rake/rdoctask'
+  RDoc::Task = Rake::RDocTask
+end
+
+RDoc::Task.new(:rdoc) do |rdoc|
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title    = 'AdsenseCrawlerForPrivate'
+  rdoc.options << '--line-numbers'
+  rdoc.rdoc_files.include('README.rdoc')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+APP_RAKEFILE = File.expand_path("../test/dummy/Rakefile", __FILE__)
+load 'rails/tasks/engine.rake'
+
+
+Bundler::GemHelper.install_tasks
+
+require 'rake/testtask'
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << 'lib'
+  t.libs << 'test'
+  t.pattern = 'test/**/*_test.rb'
+  t.verbose = false
+end
+
+
+task :default => :test

data/app/controllers/adsense_crawler_for_private/adsense_crawler_login_controller.rb

@@ -0,0 +1,39 @@
+module AdsenseCrawlerForPrivate
+  class AdsenseCrawlerLoginController < ApplicationController
+
+    # Making sure that verify_authenticity_token is not on, adsense does not have it
+    skip_before_filter :verify_authenticity_token, :only => :login
+
+    def login
+      unless AdsenseCrawlerForPrivate.crawler_password.blank?
+
+        crawler_name = params[:name]
+        crawler_password = params[:password]
+
+        if (AdsenseCrawlerForPrivate.ip_check(request) and
+            crawler_name == AdsenseCrawlerForPrivate.crawler_name and
+            crawler_password == AdsenseCrawlerForPrivate.crawler_password)
+
+          cookies.signed[AdsenseCrawlerForPrivate.cookie_name] = AdsenseCrawlerForPrivate.cookie_hash(crawler_name, crawler_password, request.remote_addr)
+
+          AdsenseCrawlerForPrivate.logger.warn "login successfully. Crawler_name: #{crawler_name}"
+
+          render :text => 'crawler login ok', :status => 200
+        else
+          cookies.delete(AdsenseCrawlerForPrivate.cookie_name, :domain => AdsenseCrawlerForPrivate.cookie_domain)
+
+          AdsenseCrawlerForPrivate.logger.warn "login unsuccessful. Crawler_name: #{crawler_name}, crawler_password: #{crawler_password}, crawler_ip: #{request.remote_addr}"
+
+          render :text => 'crawler login unsuccessful', :status => 401 # 401 unauthorized
+        end
+
+      else
+        str = "AdsenseCrawlerForPrivate not configured, no password given."
+        AdsenseCrawlerForPrivate.logger.warn(str)
+        render :text => str, :status => 401
+      end
+
+    end
+  
+  end
+end

data/app/controllers/adsense_crawler_for_private/application_controller.rb

@@ -0,0 +1,6 @@
+module AdsenseCrawlerForPrivate
+  class ApplicationController < ActionController::Base
+    
+
+  end
+end

data/app/helpers/adsense_crawler_for_private/application_helper.rb

@@ -0,0 +1,4 @@
+module AdsenseCrawlerForPrivate
+  module ApplicationHelper
+  end
+end

data/config/routes.rb

@@ -0,0 +1,3 @@
+AdsenseCrawlerForPrivate::Engine.routes.draw do
+  get "login", :controller => "adsense_crawler_login"
+end

data/lib/adsense_crawler_for_private/engine.rb

@@ -0,0 +1,6 @@
+module AdsenseCrawlerForPrivate
+  class Engine < Rails::Engine
+    isolate_namespace AdsenseCrawlerForPrivate
+
+  end
+end

data/lib/adsense_crawler_for_private/version.rb

@@ -0,0 +1,3 @@
+module AdsenseCrawlerForPrivate
+  VERSION = "1.0.0"
+end

data/lib/adsense_crawler_for_private.rb

@@ -0,0 +1,84 @@
+require 'ipaddr'
+require "adsense_crawler_for_private/engine"
+
+module AdsenseCrawlerForPrivate
+
+  mattr_accessor :cookie_name, :cookie_domain, :crawler_name, :crawler_password, :ip_ranges
+
+  protected :crawler_password
+
+  # Checks crawler cookie, returns true if logged in
+	def self.login_check(cookies, request)
+    cookie = cookies.signed[AdsenseCrawlerForPrivate.cookie_name]
+
+    login_ok = false
+
+    if !cookie.blank?
+      self.logger.info "login_check: cookie found #{cookie}"
+      begin
+        name, password, expiry_time, remote_addr = JSON.parse(cookie)
+        expiry_time = Time.parse(expiry_time)
+
+        if (name == AdsenseCrawlerForPrivate.crawler_name and
+            password == Digest::SHA1.hexdigest(AdsenseCrawlerForPrivate.crawler_password) and
+            expiry_time > Time.now and
+            request.remote_addr == remote_addr and
+            self.ip_check(request))
+          login_ok = true
+          self.logger.warn "login_check was ok for #{name}"
+        end
+      rescue JSON::ParserError => e
+        self.logger.warn "login_check problem parsing cookie json: #{e.inspect}"
+      ensure
+        unless login_ok
+          self.logger.warn "login_check wasn't ok, even though cookie was found."
+          cookies.delete(AdsenseCrawlerForPrivate.cookie_name)
+        end
+      end
+    end
+    
+    return login_ok
+  end
+
+  def self.cookie_hash(crawler_name, crawler_password, request_or_ip)
+    {:value => AdsenseCrawlerForPrivate.cookie_str(crawler_name, crawler_password, 2.days.from_now, request_or_ip),
+     :expires => 2.days.from_now,
+     :domain => AdsenseCrawlerForPrivate.cookie_domain}
+  end
+
+	def self.cookie_str(crawler_name, crawler_password, expire_time, request_or_ip)
+    ip_str = request_or_ip.respond_to?(:remote_addr) ? request_or_ip.remote_addr : request_or_ip.to_s
+
+    return [crawler_name, Digest::SHA1.hexdigest(crawler_password),
+            expire_time.httpdate, ip_str].to_json
+	end
+
+	def self.ip_check(request)
+    unless AdsenseCrawlerForPrivate.ip_ranges.nil?
+      ip_check = ::IPAddr.new(request.remote_addr)
+      AdsenseCrawlerForPrivate.ip_ranges.each do |ip_accepted|
+        return true if ip_accepted.include?(ip_check)
+      end
+      return false
+    end
+    return true
+  end
+
+  def self.logger
+    Logger
+  end
+
+  class Logger
+    def self.info(str)
+      Rails.logger.info("AdsenseCrawlerForPrivate: #{str}")
+    end
+    def self.warn(str)
+      Rails.logger.warn("AdsenseCrawlerForPrivate: #{str}")
+    end
+  end
+end
+
+#class ApplicationController < ActionController::Base
+  #   helper MyEngine::SharedEngineHelper
+  # end
+#end

data/lib/tasks/adsense_crawler_for_private_tasks.rake

@@ -0,0 +1,4 @@
+# desc "Explaining what the task does"
+# task :adsense_crawler_for_private do
+#   # Task goes here
+# end

data/test/adsense_crawler_for_private_test.rb

@@ -0,0 +1,7 @@
+require 'test_helper'
+
+class AdsenseCrawlerForPrivateTest < ActiveSupport::TestCase
+  test "truth" do
+    assert_kind_of Module, AdsenseCrawlerForPrivate
+  end
+end

data/test/dummy/Rakefile

@@ -0,0 +1,7 @@
+#!/usr/bin/env rake
+# Add your own tasks in files placed in lib/tasks ending in .rake,
+# for example lib/tasks/capistrano.rake, and they will automatically be available to Rake.
+
+require File.expand_path('../config/application', __FILE__)
+
+Dummy::Application.load_tasks

data/test/dummy/app/controllers/application_controller.rb

@@ -0,0 +1,3 @@
+class ApplicationController < ActionController::Base
+  protect_from_forgery
+end

data/test/dummy/app/controllers/main_controller.rb

@@ -0,0 +1,29 @@
+class MainController < ApplicationController
+
+  before_filter :adsense_crawler_private_specific_page, :only => :forbidden_render
+  before_filter :redirect_if_not_logged, :only => :forbidden_render
+
+
+  def normal_render
+    #debugger
+    render :text => 'this is rendered normally'
+  end
+
+  def forbidden_render
+    render :text => 'never used'
+  end
+
+  protected
+
+  def redirect_if_not_logged
+    redirect_to "http://www.you-were-not-logged-in.inv"
+  end
+
+  def adsense_crawler_private_specific_page
+    if AdsenseCrawlerForPrivate.login_check(cookies, request)
+      render :text => "private render for crawler"
+      return false
+    end
+  end
+
+end

data/test/dummy/app/helpers/application_helper.rb

@@ -0,0 +1,2 @@
+module ApplicationHelper
+end

data/test/dummy/config/application.rb

@@ -0,0 +1,50 @@
+require File.expand_path('../boot', __FILE__)
+
+# This is a custom way of disable activerecord database, used instead of default "require 'rails/all'"
+# see http://stackoverflow.com/questions/2212709/remove-activerecord-in-rails-3-beta for more info
+require "action_controller/railtie"
+require "action_mailer/railtie"
+require "active_resource/railtie"
+require "sprockets/railtie"
+
+Bundler.require
+require "adsense_crawler_for_private"
+
+module Dummy
+  class Application < Rails::Application
+    # Settings in config/environments/* take precedence over those specified here.
+    # Application configuration should go into files in config/initializers
+    # -- all .rb files in that directory are automatically loaded.
+
+    # Custom directories with classes and modules you want to be autoloadable.
+    # config.autoload_paths += %W(#{config.root}/extras)
+
+    # Only load the plugins named here, in the order given (default is alphabetical).
+    # :all can be used as a placeholder for all plugins not explicitly named.
+    # config.plugins = [ :exception_notification, :ssl_requirement, :all ]
+
+    # Activate observers that should always be running.
+    # config.active_record.observers = :cacher, :garbage_collector, :forum_observer
+
+    # Set Time.zone default to the specified zone and make Active Record auto-convert to this zone.
+    # Run "rake -D time" for a list of tasks for finding time zone names. Default is UTC.
+    # config.time_zone = 'Central Time (US & Canada)'
+
+    # The default locale is :en and all translations from config/locales/*.rb,yml are auto loaded.
+    # config.i18n.load_path += Dir[Rails.root.join('my', 'locales', '*.{rb,yml}').to_s]
+    # config.i18n.default_locale = :de
+
+    # Configure the default encoding used in templates for Ruby 1.9.
+    config.encoding = "utf-8"
+
+    # Configure sensitive parameters which will be filtered from the log file.
+    config.filter_parameters += [:password]
+
+    # Enable the asset pipeline
+    config.assets.enabled = true
+
+    # Version of your assets, change this if you want to expire all your assets
+    config.assets.version = '1.0'
+  end
+end
+