hashbang 0.0.1.alpha → 1.0.0.beta2

data/Gemfile

@@ -0,0 +1,3 @@
+source "http://rubygems.org"
+
+gemspec

data/Gemfile.lock

@@ -0,0 +1,28 @@
+PATH
+  remote: .
+  specs:
+    hashbang (0.0.1.alpha)
+      headless
+      sunscraper (~> 1.1.0.beta3)
+      unicorn
+
+GEM
+  remote: http://rubygems.org/
+  specs:
+    ffi (1.0.11)
+    headless (0.2.2)
+    kgio (2.7.2)
+    rack (1.4.1)
+    raindrops (0.8.0)
+    sunscraper (1.1.0.beta3)
+      ffi (>= 1.0.11)
+    unicorn (4.2.0)
+      kgio (~> 2.6)
+      rack
+      raindrops (~> 0.7)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  hashbang!

data/README.md

@@ -1,9 +1,9 @@
 # Bang. Hashbang.
 
-Hashbang will automatically enable serving HTML dumps for your AJAX web-applications according to 
-[Google conventions](http://code.google.com/web/ajaxcrawling/). Make your Rails AJAX applications indexable in no time.
+Hashbang is a tiny Rack proxy serving HTML dumps for your RICH web-applications according to 
+[Google AJAX Crawling conventions](http://code.google.com/web/ajaxcrawling/). Make your Rails AJAX applications indexable in no time.
 
-Hashbang will make all incoming requests containing magic `_escaped_fragment_` parameter to be served in a special way. This case will cause a subrequest to a real AJAX URL using virtual browser. This hidden browser will wait for some condition you define using well-known [Watir](http://watirwebdriver.com/) API. And then return an HTML dump.
+Using Rails generators Hashbang will setup small inner Rack application which will handle all magic requests containing `_escaped_fragment_` parameter. These requests will cause a subrequest to a real AJAX URL using virtual browser. This hidden browser will wait for some condition you define using well-known [Watir](http://watirwebdriver.com/) API. And then return an HTML dump.
 
 Let's say for example you've got a request to `test.com/?_escaped_fragment_=/my_hidden_page`.
 
@@ -11,24 +11,28 @@ Hashbang will convert this URL to `test.com/#!/my_hidden_page` and open it in a
 
 Virtual browser will call your lambda with `browser` object as parameter. With help of this lambda you can setup the wait behavior. Here is a great introduction to [Watir wait API](http://watirwebdriver.com/waiting/). Note that your lambda will act as a block to `Watir::Wait.until`.
 
-## Installation & Example
+## Installation
 
 Start from your Gemfile:
 
 ```
-  gem 'hashbang'
+gem 'hashbang'
 ```
-  
-Waiter can be defined inside your environments:
 
-```ruby
-  config.hashbang.waiter = -> b { b.execute_script("return Joosy.Application.loading") == false }
+And follow with basic generator:
+
+```
+rails g hashbang
 ```
 
-This code will wait for javascript Joosy.Application.loading variable to be set to false.
+This generator will create an inline Rack application at `hashbang/` dir. To set lambda you want to use to check if your page is ready refer to `hashbang/config.rb`.
+
+## Environments are specific
+
+While working at development environment, this gem will catch all the requests directly from rails using middleware and therefore it will just work (see P.S. below :). However due to security and performance reasons, at the production servers you are supposed to boot this Rack app separately and manually forward all magic requests to it. We'll describe typical production nginx/passenger setup later in this README.
 
-## Development environment
+P.S. Since basic development setup will use just one Rails instance in most cases all the requests to magic urls will lead to Deadlock! To solve this problem we've included the `rake hashbang:rails` command which will run your Rails project inside a [Unicorn](http://unicorn.bogomips.org/) with 2 instances.
 
-By its nature Hashbang gem requires your server to accept at least 2 concurrent connections. However WEBRick or Thin (which are mostly used as development servers) will only serve one Rails instance (and connection). That's why if you want to debug your AJAX crawling, you'll have to use `unicorn` with at least 2 workers.
+## Memory consumption
 
-Note either that browser load behavior differs between development and production environments. Hashbang will use lazy load for development and therefore it may take some time for crawler to respond. While running in production environment it will start and cache browser instance at rails startup to provide the best possible response speed.
+This part of hashbang is currently in progress. We still bundle Watir chromedev for proof-of-concept reasons. The full version will come with Qt WebKit bindings.

data/hashbang.gemspec

@@ -1,16 +1,17 @@
 Gem::Specification.new do |s|
   s.name        = "hashbang"
-  s.version     = "0.0.1.alpha"
+  s.version     = "1.0.0.beta2"
   s.platform    = Gem::Platform::RUBY
   s.summary     = "Magic support of Google/Bing/... AJAX search indexing for your Rails apps"
   s.email       = "boris@roundlake.ru"
   s.homepage    = "http://roundlake.github.com/hashbang/"
-  s.description = "Hashbang will automatically enable serving HTML dumps for your AJAX web-applications according to Google conventions. Make your Rails AJAX applications indexable in no time."
+  s.description = "Hashbang is a tiny Rack proxy serving HTML dumps for your RICH web-applications according to Google AJAX Crawling conventions. Make your Rails AJAX applications indexable in no time."
   s.authors     = ['Boris Staal']
 
   s.files         = `git ls-files`.split("\n")
   s.require_paths = ["lib"]
 
-  s.add_dependency 'watir-webdriver'
   s.add_dependency 'headless'
+  s.add_dependency 'sunscraper', '~> 1.1.0.beta3'
+  s.add_dependency 'unicorn'
 end

data/lib/generators/hashbang_generator.rb

@@ -0,0 +1,16 @@
+class HashbangGenerator < Rails::Generators::Base
+  source_root File.join(File.dirname(__FILE__), 'templates')
+
+  def create_files
+    empty_directory "hashbang"
+    template "config.ru", "hashbang/config.ru"
+    template "config.rb", "hashbang/config.rb"
+    template "unicorn.rb", "hashbang/unicorn.rb"
+
+    empty_directory "hashbang/tmp"
+    create_file "hashbang/tmp/.gitkeep"
+
+    empty_directory "hashbang/public"
+    create_file "hashbang/public/.gitkeep"
+  end
+end

data/lib/generators/templates/config.rb

@@ -0,0 +1,4 @@
+Hashbang::Config.map do |c|
+  c.url     = /^$/
+  c.timeout = 5000
+end

data/lib/generators/templates/config.ru

@@ -0,0 +1,11 @@
+ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../../Gemfile', __FILE__)
+
+require 'rubygems'
+require 'bundler'
+
+$: << Bundler.load.specs.find{|s| s.name == 'hashbang' }.full_gem_path + '/lib'
+
+require 'hashbang'
+require 'hashbang/standalone/middleware'
+
+run Hashbang::Standalone::Middleware.new(File.expand_path('..',  __FILE__))

data/lib/generators/templates/unicorn.rb

@@ -0,0 +1,3 @@
+preload_app true
+worker_processes 2
+listen 3000

data/lib/hashbang.rb

@@ -1,7 +1,6 @@
 require 'headless'
-require 'watir-webdriver'
+require 'sunscraper'
 
-require 'hashbang/pool'
 require 'hashbang/crawler'
-require 'hashbang/middleware'
-require 'hashbang/railtie' if defined? Rails
+require 'hashbang/config'
+require 'hashbang/railtie/engine' if defined? Rails

data/lib/hashbang/config.rb

@@ -0,0 +1,19 @@
+module Hashbang
+  module Config
+    attr_accessor :url, :timeout
+
+    extend self
+
+    def map
+      yield self
+    end
+
+    def load(path)
+      if File.exists? path
+        require path
+      end
+
+      self.url = /^#{url}/ unless self.url.is_a? Regexp
+    end
+  end
+end

data/lib/hashbang/crawler.rb

@@ -1,20 +1,24 @@
 module Hashbang
-  class Crawler
-    def self.gimme(url, &block)
-      browser = Pool.grab
+  module Crawler
+    extend self
 
-      begin
-        browser.goto url
+    def gimme(url, timeout)
+      Sunscraper.scrape_url url, timeout
+    end
+
+    def urlFromRack(environment)
+      url = []
+      url << environment['rack.url_scheme'] + '://'
+      url << environment['HTTP_HOST']
+      url << environment['REQUEST_PATH']
+      url << '?' unless environment['QUERY_STRING'].starts_with? '_escaped_fragment_'
+      url << environment['QUERY_STRING'].gsub(/(\&)?_escaped_fragment_=/, '#!')
 
-        if block_given?
-          Watir::Wait.until { block.call browser }
-        end
+      url.join ''
+    end
 
-        browser.html
-      rescue
-      ensure
-        Hashbang::Pool.release(browser)
-      end
+    def urlFromUrl(url)
+      url.gsub(/[\?\&]_escaped_fragment_=/, '#!')
     end
   end
 end

data/lib/hashbang/railtie/engine.rb

@@ -0,0 +1,18 @@
+require 'hashbang/railtie/middleware'
+
+module Hashbang
+  module Railtie
+    class Engine < Rails::Engine
+      initializer "application_controller.initialize_hashbang" do |app|
+        if Rails.env == 'development'
+          app.config.middleware.use "Hashbang::Railtie::Middleware"
+          Config::load Rails.root.join('hashbang/config.rb').to_s
+        end
+      end
+
+      rake_tasks do
+        Dir[File.expand_path('../../tasks/*.rake',  __FILE__)].each { |f| load f }
+      end
+    end
+  end
+end

data/lib/hashbang/railtie/middleware.rb

@@ -0,0 +1,21 @@
+module Hashbang
+  module Railtie
+    class Middleware
+      def initialize(application)
+        @application = application
+        Headless.new.start
+      end
+
+      def call(environment)
+        if environment['QUERY_STRING'].include? "_escaped_fragment_"
+          url  = Crawler.urlFromRack(environment)
+          html = Crawler.gimme url, 100000
+
+          [200, {"Content-Type" => "text/html; charset=utf-8"}, [html]]
+        else
+          @application.call(environment)
+        end
+      end
+    end
+  end
+end

data/lib/hashbang/standalone/middleware.rb

@@ -0,0 +1,33 @@
+# coding: utf-8
+module Hashbang
+  module Standalone
+    class Middleware
+      def initialize(root)
+        Config.load File.expand_path('config.rb', root)
+        Headless.new.start
+      end
+
+      def call(environment)
+        url = environment['QUERY_STRING'].split('&').find{|x| x[0,4] == 'url='}
+
+        unless url.to_s.length == 0
+          url = url.split('=')[1]
+          url = URI.unescape url
+          url = Crawler.urlFromUrl url
+        end
+
+        if url.to_s.length == 0 || !url.match(Config.url)
+          return [200, {"Content-Type" => "text/html; charset=utf-8"}, ['']]
+        end
+
+        html = Crawler.gimme url, Config.timeout
+
+        if html.respond_to? :force_encoding
+          html.force_encoding "UTF-8"
+        end
+
+        return [200, {"Content-Type" => "text/html; charset=utf-8"}, [html]]
+      end
+    end
+  end
+end

data/lib/tasks/hashbang.rake

@@ -0,0 +1,24 @@
+namespace :hashbang do
+
+  desc "Start development unicorn"
+  task :rails do
+    config = "#{Dir.getwd}/hashbang/unicorn.rb"
+
+    unless File.exists? config
+      raise "Hashbang subdirectory does not seem to exist. Did you run `rails g hashbang`?"
+    end
+
+    sh "bundle exec unicorn --config-file #{config}"
+  end
+
+  desc "Start production standaloner"
+  task :standalone do
+    rackup = "#{Dir.getwd}/hashbang/config.ru"
+
+    unless File.exists? rackup
+      raise "Hashbang subdirectory does not seem to exist. Did you run `rails g hashbang`?"
+    end
+
+    sh "bundle exec rackup #{rackup}"
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: hashbang
 version: !ruby/object:Gem::Version
-  version: 0.0.1.alpha
+  version: 1.0.0.beta2
   prerelease: 6
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-02-15 00:00:00.000000000 Z
+date: 2012-03-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: watir-webdriver
-  requirement: &70179137315820 !ruby/object:Gem::Requirement
+  name: headless
+  requirement: &70344278924840 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,21 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70179137315820
+  version_requirements: *70344278924840
 - !ruby/object:Gem::Dependency
-  name: headless
-  requirement: &70179137315360 !ruby/object:Gem::Requirement
+  name: sunscraper
+  requirement: &70344278922420 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.1.0.beta3
+  type: :runtime
+  prerelease: false
+  version_requirements: *70344278922420
+- !ruby/object:Gem::Dependency
+  name: unicorn
+  requirement: &70344278920940 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,24 +43,32 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70179137315360
-description: Hashbang will automatically enable serving HTML dumps for your AJAX web-applications
-  according to Google conventions. Make your Rails AJAX applications indexable in
-  no time.
+  version_requirements: *70344278920940
+description: Hashbang is a tiny Rack proxy serving HTML dumps for your RICH web-applications
+  according to Google AJAX Crawling conventions. Make your Rails AJAX applications
+  indexable in no time.
 email: boris@roundlake.ru
 executables: []
 extensions: []
 extra_rdoc_files: []
 files:
+- Gemfile
+- Gemfile.lock
 - README.md
 - Rakefile
 - hashbang.gemspec
 - init.rb
+- lib/generators/hashbang_generator.rb
+- lib/generators/templates/config.rb
+- lib/generators/templates/config.ru
+- lib/generators/templates/unicorn.rb
 - lib/hashbang.rb
+- lib/hashbang/config.rb
 - lib/hashbang/crawler.rb
-- lib/hashbang/middleware.rb
-- lib/hashbang/pool.rb
-- lib/hashbang/railtie.rb
+- lib/hashbang/railtie/engine.rb
+- lib/hashbang/railtie/middleware.rb
+- lib/hashbang/standalone/middleware.rb
+- lib/tasks/hashbang.rake
 homepage: http://roundlake.github.com/hashbang/
 licenses: []
 post_install_message: 

data/lib/hashbang/middleware.rb

@@ -1,35 +0,0 @@
-module Hashbang
-  class Middleware
-    def initialize(application, config)
-      @application = application
-      @config      = config
-
-      Headless.new.start
-      Pool.setup config.pool_size
-    end
-
-    def call(environment)
-      if environment['QUERY_STRING'].include? "_escaped_fragment_"
-        url = []
-        url << environment['rack.url_scheme'] + '://'
-        url << environment['HTTP_HOST']
-        url << environment['REQUEST_PATH']
-        url << '?' unless environment['QUERY_STRING'].starts_with? '_escaped_fragment_'
-        url << environment['QUERY_STRING'].gsub(/(\&)?_escaped_fragment_=/, '#!')
-        url = url.join ''
-
-        if @config.waiter
-          html = Crawler.gimme(url) do |browser|
-            @config.waiter.call(browser)
-          end
-        else
-          html = Crawler.gimme url
-        end
-
-        [200, {"Content-Type" => "text/html"}, [html]]
-      else
-        @application.call(environment)
-      end
-    end
-  end
-end

data/lib/hashbang/pool.rb

@@ -1,40 +0,0 @@
-module Hashbang
-  class Pool
-    cattr_accessor :inited
-    cattr_accessor :pool_size
-    cattr_accessor :browsers
-    cattr_accessor :taken
-
-    def self.setup(quantity=1)
-      @@pool_size = quantity
-
-      self.init if !Rails || Rails.env != 'development'
-    end
-
-    def self.init
-      @@browsers = []
-      @@taken    = []
-
-      @@pool_size.times do
-        @@browsers << browser = Watir::Browser.new
-        at_exit { browser.close if browser.exists? }
-      end
-
-      @@inited = true
-    end
-
-    def self.grab
-      self.init unless @@inited
-
-      raise "Pool is empty" if @@browsers.length == 0
-
-      @@taken << browser = @@browsers.pop
-      browser
-    end
-
-    def self.release(browser)
-      @@taken.delete browser
-      @@browsers << browser
-    end
-  end
-end

data/lib/hashbang/railtie.rb

@@ -1,20 +0,0 @@
-module Hashbang
-  class Railtie < Rails::Railtie
-    config.hashbang = ActiveSupport::OrderedOptions.new
-
-    config.hashbang.pool_size = 1
-    config.hashbang.waiter    = false
-
-    waiters = {
-      :joosy => -> b { b.execute_script("return Joosy.Application.loading") == false }
-    }
-
-    initializer "application_controller.initialize_hashbang" do |app|
-      if app.config.hashbang.waiter.is_a? Symbol
-        app.config.hashbang.waiter = waiters[app.config.hashbang.waiter]
-      end
-
-      app.config.middleware.use "Hashbang::Middleware", app.config.hashbang
-    end
-  end
-end