hashbang 1.0.0.beta2 → 1.0.0

data/README.md

@@ -3,17 +3,37 @@
 Hashbang is a tiny Rack proxy serving HTML dumps for your RICH web-applications according to 
 [Google AJAX Crawling conventions](http://code.google.com/web/ajaxcrawling/). Make your Rails AJAX applications indexable in no time.
 
-Using Rails generators Hashbang will setup small inner Rack application which will handle all magic requests containing `_escaped_fragment_` parameter. These requests will cause a subrequest to a real AJAX URL using virtual browser. This hidden browser will wait for some condition you define using well-known [Watir](http://watirwebdriver.com/) API. And then return an HTML dump.
+Using Rails generators Hashbang will setup a small inner Rack application which will handle all magic requests containing `_escaped_fragment_` parameter. These requests will cause a subrequest to a real AJAX URL using a virtual browser. 
 
-Let's say for example you've got a request to `test.com/?_escaped_fragment_=/my_hidden_page`.
+Let's say for example you've got a request to `test.com/?_escaped_fragment_=/my_hidden_page`. Hashbang will convert this URL to `test.com/#!/my_hidden_page` and open it in the virtual browser. The virtual browser will load this page and wait for `Suncscraper.finish` javascript call. As soon as it was called Hashbang will respond with an HTML dump.
 
-Hashbang will convert this URL to `test.com/#!/my_hidden_page` and open it in a virtual browser.
+Hashbang uses [Sunscraper](http://github.com/roundlake/sunscraper) and therefore you will need Qt to use it.
 
-Virtual browser will call your lambda with `browser` object as parameter. With help of this lambda you can setup the wait behavior. Here is a great introduction to [Watir wait API](http://watirwebdriver.com/waiting/). Note that your lambda will act as a block to `Watir::Wait.until`.
+## Environments are specific
+
+If you are not using Rails you should skip this paragraph. While working with at development environment, this gem will catch all the requests containing `_escaped_fragment_` directly from Rails using middleware and therefore it will just work **(see P.S. below)**. Go to `http://localhost:3000?_escaped_fragment_=test` to make Hashbang load and dump `http://localhost:3000/#!/test` for you.
+
+Due to security and performance reasons, at the production servers you are supposed to boot this Rack app separately and manually forward all the magic requests to the standalone instance.
+
+Imagine you are runing Hashbang rack instance at `33222` port. With that you should proxy all the requests containing `_escaped_fragment_=` to `localhost:33222/?url=…` where `…` is a full request URI. **Don't forget to escape url parameter so resulting request could be like this:** `localhost:33222/?url=http%3A%2F%2Fwww.dvnts.ru%2F%3F_encoded_fragment_%3D`.
+
+You are supposed to limit the concurent connections and restrict the direct connection to Hashbang instances. We'll describe typical production nginx/passenger setup later in this README.
+
+**P.S.**
+
+Since in most cases basic development setup uses just one Rails instance, all the requests to magic urls will lead to Deadlock! To solve this problem we've included the `rake hashbang:rails` command which will run your Rails project inside a [Unicorn](http://unicorn.bogomips.org/) with 2 instances.
+
+You can also simulate production mode runing `rake hashbang:standalone`.
 
 ## Installation
 
-Start from your Gemfile:
+Ensure you have the Qt dependencies for Sunscraper (read [Sunscraper description](http://github.com/roundlake/sunscraper/) for more info).
+
+To install it on Mac with [Homebrew](http://mxcl.github.com/homebrew/) run `brew install qt`. 
+
+To install it on Debian run `apt-get install qt4-dev-tools --no-install-recommends`.
+
+Add gem to your Gemfile:
 
 ```
 gem 'hashbang'
@@ -25,14 +45,33 @@ And follow with basic generator:
 rails g hashbang
 ```
 
-This generator will create an inline Rack application at `hashbang/` dir. To set lambda you want to use to check if your page is ready refer to `hashbang/config.rb`.
+This generator will create an inline Rack application at `hashbang/` dir. You can proceed with required configuration at `hashbang/config.rb`.
 
-## Environments are specific
+## Configuration
+
+```ruby
+Hashbang::Config.map do |c|
+  c.url     = /^.*$/
+  c.timeout = 5000
+end
+```
+
+#### Url
 
-While working at development environment, this gem will catch all the requests directly from rails using middleware and therefore it will just work (see P.S. below :). However due to security and performance reasons, at the production servers you are supposed to boot this Rack app separately and manually forward all magic requests to it. We'll describe typical production nginx/passenger setup later in this README.
+Url limits hashbang crawling to described set of URLs. The limit only works in standalone mode.
 
-P.S. Since basic development setup will use just one Rails instance in most cases all the requests to magic urls will lead to Deadlock! To solve this problem we've included the `rake hashbang:rails` command which will run your Rails project inside a [Unicorn](http://unicorn.bogomips.org/) with 2 instances.
+#### Timeout
+
+Hashbang will give virtual browser specified timeout in miliseconds to grab data. Keep it as low as possible. The timeout only works in standalone mode.
+
+## Crawling marker
+
+To help Sunscraper (virtual browser of Hashbang) understand what should be considered a loaded page, add Javascript `Suncscraper.finish()` call when all AJAX is done and your DOM is ready. Note that for the straight client calls `Sunscraper` variable will be empty and therefore you should check if it's available. This is how it should basically look:
+
+```javascript
+if (typeof Sunscraper !== "undefined") { Sunscraper.finish() }
+```
 
 ## Memory consumption
 
-This part of hashbang is currently in progress. We still bundle Watir chromedev for proof-of-concept reasons. The full version will come with Qt WebKit bindings.
+Hashbang will keep one instance of Sunscraper per each Hashbang instance. Sunscraper  bundles clear QTWebKit and therefore keeps memory consumption as low as possible for virtual browsers. However it can still be noticeable and therefore you should only increase possible concurency if your resource gets indexed often.

data/hashbang.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name        = "hashbang"
-  s.version     = "1.0.0.beta2"
+  s.version     = "1.0.0"
   s.platform    = Gem::Platform::RUBY
   s.summary     = "Magic support of Google/Bing/... AJAX search indexing for your Rails apps"
   s.email       = "boris@roundlake.ru"
@@ -13,5 +13,5 @@ Gem::Specification.new do |s|
 
   s.add_dependency 'headless'
   s.add_dependency 'sunscraper', '~> 1.1.0.beta3'
-  s.add_dependency 'unicorn'
+  s.add_development_dependency 'unicorn'
 end

data/lib/generators/templates/config.rb

@@ -1,4 +1,4 @@
 Hashbang::Config.map do |c|
-  c.url     = /^$/
+  c.url     = /^.*$/
   c.timeout = 5000
 end

metadata

@@ -1,8 +1,8 @@
 --- !ruby/object:Gem::Specification
 name: hashbang
 version: !ruby/object:Gem::Version
-  version: 1.0.0.beta2
-  prerelease: 6
+  version: 1.0.0
+  prerelease: 
 platform: ruby
 authors:
 - Boris Staal
@@ -13,7 +13,7 @@ date: 2012-03-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: headless
-  requirement: &70344278924840 !ruby/object:Gem::Requirement
+  requirement: &70247960918140 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70344278924840
+  version_requirements: *70247960918140
 - !ruby/object:Gem::Dependency
   name: sunscraper
-  requirement: &70344278922420 !ruby/object:Gem::Requirement
+  requirement: &70247960915780 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -32,18 +32,18 @@ dependencies:
         version: 1.1.0.beta3
   type: :runtime
   prerelease: false
-  version_requirements: *70344278922420
+  version_requirements: *70247960915780
 - !ruby/object:Gem::Dependency
   name: unicorn
-  requirement: &70344278920940 !ruby/object:Gem::Requirement
+  requirement: &70247960913160 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
       - !ruby/object:Gem::Version
         version: '0'
-  type: :runtime
+  type: :development
   prerelease: false
-  version_requirements: *70344278920940
+  version_requirements: *70247960913160
 description: Hashbang is a tiny Rack proxy serving HTML dumps for your RICH web-applications
   according to Google AJAX Crawling conventions. Make your Rails AJAX applications
   indexable in no time.
@@ -84,9 +84,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
-  - - ! '>'
+  - - ! '>='
     - !ruby/object:Gem::Version
-      version: 1.3.1
+      version: '0'
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.15