shrimple 0.8.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: e11086eed094342c573b7e63b6150d44ceb089ba
+  data.tar.gz: 73c918822c7f614950bc832aa606e56a19f14392
+SHA512:
+  metadata.gz: ec92531f5eb31c9546aaf6406956cfaee57dd55188f41ef7e800644e755d16728c2c8dbae9df18952a7ebc4281b3a92761e8d9e7eb98076064a963b99e06aa24
+  data.tar.gz: 9f0263d4910fbcc6a2f5873fafb18707f837da080d30390d6ad7909d3bdac6fe0f0efa86636decc004bac1a9cc8554a4a5cba5572bda158390a203045b72c63e

data/.gitignore

@@ -0,0 +1,2 @@
+*.gem
+Gemfile.lock

data/.travis.yml

@@ -0,0 +1,9 @@
+language: ruby
+
+rvm:
+  - 1.9.3
+  - 2.0.0
+  - 2.1.1
+  - ruby-head
+  - jruby-19mode
+  - jruby-head

data/Gemfile

@@ -0,0 +1,9 @@
+source 'https://rubygems.org'
+
+gem 'hashie'
+
+group :test do
+  gem 'rspec'
+  gem 'rake'  # for travis-ci
+  gem 'dimensions'
+end

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2012 adeven GmbH Manuel Kniep
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,125 @@
+# Shrimple
+
+Launches PhantomJS to render web sites or local files (or have
+Phantom do pretty much everything).
+Shrimple started as a set of patches for [Shrimp](https://github.com/adjust/shrimp).
+
+[![Build Status](https://travis-ci.org/bronson/shrimple.svg?branch=master)](https://travis-ci.org/bronson/shrimple)
+[![Gem Version](https://badge.fury.io/rb/shrimple.svg)](http://badge.fury.io/rb/shrimple)
+
+
+
+## Installation
+
+Install [PhantomJS](http://phantomjs.org/download.html), then add this line to your application's Gemfile:
+
+    gem 'shrimple', git: 'https://github.com/bronson/shrimple'
+
+and execute `bundle`.
+
+## Usage
+
+Render to a file:
+
+```ruby
+require 'shrimple'
+
+s = Shrimple.new( page: { paperSize: { format: 'A4' }} )
+s.render_pdf('http://bl.ocks.org/mbostock', to: '/tmp/output.pdf')
+```
+
+Or render to a variable by omitting the destination:
+
+```ruby
+result = Shrimple.new.render_text('http://thingsididlastnight.com')
+result.stdout   # <== TODO: naming stdout is arcane
+=> "Your Mom\n"
+```
+
+Render in the background:
+
+TODO: show background mode
+TODO: show start_time and finish_time?
+
+
+## Configuration
+
+Shrimple supports all configuration options provided by PhantomJS
+(including unanticipated ones added in the future).
+
+Options specified later override those specified earlier, and
+options passed directly to render only affect that call -- they are not remembered.
+
+```ruby
+s = Shrimple.new( page: { zoomFactor: 0.5 }, timeout: 10 )
+s.page.paperSize = { border: '3cm', format: 'A4', orientation: 'landscape' }
+s.render_pdf('http://joeyh.name/blog/', to: '/tmp/joey.pdf')
+```
+
+* Set options passed to PhantomJS's [command line](http://phantomjs.org/api/command-line.html) with `config`:<br>
+`s.config.loadImages = false`<br>
+Phantom requires these to be in JSON notation: `proxyType` instead of `--proxy-type`.
+
+* Set options in PhantomJS's [web page module](http://phantomjs.org/api/webpage/) with `page`:<br>
+`s.page.paperSize.orientation = 'landscape'`
+
+* Set options passed to PhantomJS's [render call](http://phantomjs.org/api/webpage/method/render.html) with `render`:<br>
+`s.render = { format: 'jpeg', quality: 85 }`
+
+See [default_config.rb](https://github.com/bronson/shrimple/blob/master/lib/shrimple/default_config.rb)
+for the known options all listed in one place.
+
+### Shrimple Options
+
+- **background** If true, the PhantomJS process will be spawned in the background
+  and Ruby execution will resume immediatley.<br>
+  `background: false`
+
+- **timeout** The time in seconds after which the PhantomJS executable is killed.
+  If killed, the render results in an error.<br>
+  `timeout: nil`
+
+- **output / to** Specifies the destination file.  If you don't specify a destination
+  then the output is buffered into memory and can be retrieved with `result.stdout`.
+  `to` is just a more readable synonym for `output`.
+
+- **stderr** The path to save phantom's stderr.  Normally it's buffered into memory
+  and can be retrieved with `result.stderr`
+
+- **onSuccess** A Ruby proc to be called when the render succeeds.<br>
+  `onSuccess = ->(result) { ftp.put(result.stdout) }`
+
+- **onError** A Ruby proc called when the render fails or is killed.<br>
+  `onError = ->(result) { page_admin(result.stderr, result.options.to_hash) }`
+
+####
+
+These are more obscure, only necessary if you're trying to use Phantom in
+an obscure way.
+
+- **input** specifies the source file to render.  Normally you'd pass this as the first
+  argument to render.  Use this option if you want to specify the input file once and render it multiple times.
+  You must specify a valid URL.  Use `file://test_file.html` to specify a file on the local filesystem.
+
+- **execuatable** a path to the phantomjs exectuable to use.  Shrimple searches
+  pretty hard for installed phantomjs executables so there's usually no need
+  to specify this.
+
+- **renderer** the render.js script to pass to Phantom.  Probably only useful for testing.
+
+
+## Changes to Shrimp
+
+- Added background mode (even works in JRuby >1.7.4).
+- Allows configuring pretty much anything: proxies, userName/password, scrollPosition, jpeg quality, etc.
+- Prevents potential shell attacks by ensuring options aren't passed on the command line.
+- Better error handling.
+- Removed middleware.  In my app, background mode made it unnecessary.  Besides, I could never get it to work reliably.
+
+
+## Copyright
+
+Shrimp, the original project, is Copyright © 2012 adeven (Manuel Kniep).
+It is free software, and may be redistributed under the MIT License (see LICENSE.txt).
+
+Shrimple is also Copyright © 2013 Scott Bronson and may be redistributed under the same terms.

data/Rakefile

@@ -0,0 +1,7 @@
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec
+task test: :spec

data/lib/render.js

@@ -0,0 +1,68 @@
+// Invokes PhantomJS to render a webpage to stdout.  Config is supplied as json on stdin.
+
+/* jshint phantom: true */
+
+
+var system = require('system')
+var page = require('webpage').create()
+
+function errorHandler(msg, trace) {
+  system.stderr.writeLine(msg)
+  trace.forEach(function(item) {
+    system.stderr.writeLine(' -> ' + (item.file || item.sourceURL) + ': ' + item.line + (item.function ? ' (in function ' + item.function + ')' : ''));
+  })
+  phantom.exit(1)
+}
+
+phantom.onError = function(msg, trace) { errorHandler("PHANTOM ERROR: " + msg, trace) }
+
+page.onError = function(msg, trace) { errorHandler("PAGE ERROR: " + msg, trace) }
+
+var config = JSON.parse(system.stdin.read())
+
+for(var key in config.page) {
+  page[key] = config.page[key]
+}
+
+page.open(config.input, function (status) {
+  if (status !== 'success' /* || (statusCode !== 200 && statusCode !== null) */) {
+    system.stderr.writeLine('Unable to load ' + config.input);
+    phantom.exit(1);
+  }
+
+  if(config.render.format === 'text') {
+    system.stdout.writeLine(page.plainText)
+  } else if(config.render.format === 'html') {
+    system.stdout.writeLine(page.content)
+  } else {
+    page.render('/dev/stdout', config.render);
+  }
+
+  phantom.exit(0);
+});
+
+
+
+// if (typeof cookie_file !== 'undefined') {
+//   try {
+//     var f = fs.open(cookie_file, "r");
+//     cookies = JSON.parse(f.read());
+//     fs.remove(cookie_file)
+//   } catch (e) {
+//     // TODO: run this through regular error reporter.  just don't catch right?
+//     console.log(e);
+//   }
+  
+//   phantom.cookiesEnabled = true;
+//   phantom.cookies = cookies;
+// }
+
+
+
+// Determine the statusCode
+// page.onResourceReceived = function (resource) {
+//   if (resource.url === address) {
+//     statusCode = resource.status;
+//   }
+// };
+

data/lib/shrimple.rb

@@ -0,0 +1,120 @@
+# Keeps track of options and calls phantoimjs to run the render script.
+
+# TODO: make debug mode continually dump stderr to parent stderr, plus add debugging to render script
+# TODO: add a header and footer to the page, printheaderfooter.coffee
+# TODO: support for injectjs?   http://phantomjs.org/tips-and-tricks.html
+#       and maybe page.evaluate(function() { document.body.bgColor = 'white'; });
+# TODO: fix syntax of non-background mode, make retrieving data easier
+# TODO: allow constructor to accept multiple hashes, add merge method to merge options hashes
+# TODO: add ability to specify max number of processes in process_monitor
+# TODO: add onResourceTimeout: https://github.com/onlyurei/phantomjs/commit/fa5a3504070f86a99f11469a3b7eb17a0b005ef7
+# TODO: add cookiefile support?
+# TODO: should block if user tries to launch too man phantom processes?  Like SizedQueue.
+# TODO: wow --config=file sucks.  maybe add a way to specify cmdline args again?
+#       either that or fix phantomjs...  https://github.com/ariya/phantomjs/issues/12265 https://github.com/ariya/phantomjs/issues/11775
+# TODO: test that page.customHeaders appear in the network requests (how....?)
+
+# maybe:
+# TODO: support casperjs?
+# TODO: fill in both "can read partial" tests
+# TODO: include lots of info about page load in logfile
+# TODO: documentation!  probably using sdoc or yard?
+# TODO: possible to test margins?  printmargins.coffee
+# TODO: bl.ocks.org/mbostock: page renders before it's done downloading.  able to add arbitrary delay or js function to wait before rendering?
+
+
+require 'hashie/mash'
+require 'shrimple/phantom'
+require 'shrimple/default_config'
+
+
+class Shrimple
+  attr_accessor :options
+
+  # allows setting config options directly on this object: s.timeout = 10
+  def method_missing name, *args, &block
+    options.send(name, *args, &block)
+  end
+
+
+  def initialize opts={}
+    @options = Hashie::Mash.new(Shrimple::DefaultConfig)
+    @options.deep_merge!(opts)
+    self.executable ||= self.class.default_executable
+    self.renderer ||= self.class.default_renderer
+  end
+
+
+  # might be time to allow method_missing to handle these helpers...
+  def render_pdf src, *opts
+    render src, {render: {format: 'pdf'}}, *opts
+  end
+
+  def render_png src, *opts
+    render src, {render: {format: 'png'}}, *opts
+  end
+
+  def render_jpeg src, *opts
+    render src, {render: {format: 'jpeg'}}, *opts
+  end
+
+  def render_gif src, *opts
+    render src, {render: {format: 'gif'}}, *opts
+  end
+
+  def render_html src, *opts
+    render src, {render: {format: 'html'}}, *opts
+  end
+
+  def render_text src, *opts
+    render src, {render: {format: 'text'}}, *opts
+  end
+
+
+
+  def render src={}, *opts
+    full_opts = get_full_options(src, *opts)
+    phantom = Shrimple::Phantom.new(full_opts)
+    phantom.wait unless full_opts[:background]
+    phantom
+  end
+
+  def get_full_options src, *inopts
+    exopts = options.dup
+    onSuccess = exopts.delete(:onSuccess)
+    onError = exopts.delete(:onError)
+
+    full_opts = Shrimple.deep_dup(exopts)
+    full_opts.deep_merge!(src) if src && src.kind_of?(Hash)
+    inopts.each { |opt| full_opts.deep_merge!(opt) }
+    full_opts.merge!(input: src) if src && !src.kind_of?(Hash)
+    full_opts.merge!(output: full_opts.delete(:to)) if full_opts[:to]
+    full_opts.merge!(onSuccess: onSuccess, onError: onError)
+
+    self.class.compact!(full_opts)
+    full_opts
+  end
+
+
+  # how are these not a part of Hash?
+  def self.compact! hash
+    hash.delete_if { |k,v| v.nil? or (v.is_a?(Hash) && compact!(v).empty?) or (v.respond_to?('empty?') && v.empty?) }
+  end
+
+  def self.deep_dup hash
+    Marshal.load(Marshal.dump(hash))
+  end
+
+
+  def self.processes
+    @processes ||= Shrimple::ProcessMonitor.new
+  end
+
+  def self.default_renderer
+    File.expand_path('../render.js', __FILE__)
+  end
+  
+  def self.default_executable
+    (defined?(Bundler::GemfileError) ? `bundle exec which phantomjs` : `which phantomjs`).chomp
+  end
+end

data/lib/shrimple/default_config.rb

@@ -0,0 +1,114 @@
+# Here is a list of all the phantomjs settings that we know about.
+# Note that you can set any option you want.  It need not be listed here.
+#
+# nil means leave unset -- use Phantom's defaults
+
+
+class Shrimple
+  DefaultConfig = {
+
+    #
+    # options for launching the PhantomJS executable
+    #
+
+    background: nil,       # false blocks until page is rendered, true returns immediately
+    executable: nil,       # specifies the PhantomJS executable to use. If unspecified then Shrimple will search for one.
+    renderer: nil,         # the render script to use. Useful for testing, or if you want to do something other than rendering the page.
+    timeout: nil,          # time in seconds after which the PhantomJS process should simply be killed
+    input: nil,            # specifies the URL to request (use file:// for local assets).  Can also be specified by the optional first argument to render calls.
+    output: nil,           # path to the rendered output file, nil to buffer output in memory.  "to" is a more readable synonym: 'render url, to: file'.
+    stderr: nil,           # path to the file to receive PhantomJS's stderr, leave nil to store it in a string
+    onSuccess: nil,        # a function to call when the pdf has been successfully rendered.  called before process is removed from Shrimple.processes so, if it blocks, process table fills up.  Useful for rate limiting.
+    onError: nil,          # a function to call when the pdf has failed for whatever reason (timeout, killed, network error, etc).  Called before process is removed from Shrimple.processes.
+
+
+    #
+    # arguments passed to the PhantomJS render method  http://phantomjs.org/api/webpage/method/render.html
+    #
+
+    render: {
+      format: nil,         # format for the output file.  usually supplied by a helper (render_pdf, render_png, etc)
+      quality: nil         # only relevant to format=jpeg I think, range is 1-100.  not sure what Phantom's default is
+    },
+
+
+    #
+    # command-line options passed to PhantomJS in --config: http://phantomjs.org/api/command-line.html
+    #
+
+    config: {
+      cookiesFile: nil,        # path to the persitent cookies file
+      diskCache: nil,          # if true, caches requested assets.  Defaults to false.  See config.maxDiskCacheSize.  The cache location is not currently configurable.
+      ignoreSslErrors: nil,    # if true, SSL errors won't prevent page from being rendered.  defaults to false
+      loadImages: nil,         # load inlined images?  defaults to true.  see also page.settings.loadImages
+      localStoragePath: nil,   # directory to save LocalStorage and WebSQL content
+      localStorageQuota: nil,  # maximum size for local data
+      localToRemoteUrlAccess: nil,   # local content can initiate requests for remote assets?  Defaults to false. also see page.settings.localToRemoteUrlAccessEnabled
+      maxDiskCacheSize: nil,   # maximum size for disk cache in KB.  Also see config.diskCache.
+      outputEncoding: nil,     # sets the encoding used in the logfile.  nil means "utf8"
+      remoteDebuggerPort: nil, # starts the render script in a debug harness and listens on this port
+      remoteDebuggerAutorun: nil, # run the render script in a debugger?  defaults to false, probably never needed
+      proxy: nil,              # proxy to use in "address:port" format
+      proxyType: nil,          # type of proxy to use
+      proxyAuth: nil,          # authentication information for proxy
+      scriptEncoding: nil,     # encoding of the render script, defaults to "utf8"
+      sslProtocol: nil,        # the protocol to use for SSL connections, defaults to "SSLv3"
+      webSecurity: nil         # enable web security and forbid cross-domain XHR?  Defaults to true
+    },
+
+
+    #
+    # settings for rendering the page: http://phantomjs.org/api/webpage/
+    #
+
+    page: {
+      canGoBack: nil,          # allow javascript navigation, defaults to false
+      canGoForward: nil,       # allow javascript navigation, defaults to false
+      clipRect: {              # area to rasterize when page.render is called
+        left: nil,             # Defaults to (0,0,0,0) meaning render the entire page
+        top: nil,
+        width: nil,
+        height: nil
+      },
+      customHeaders: {       # headers added to every HTTP request.  if nil, Shrimple.DefaultHeaders is used.
+        "Accept-Encoding" => "identity" # Don't accept gzipped responses, work around https://github.com/ariya/phantomjs/issues/10930
+      },
+      # event?  http://phantomjs.org/api/webpage/property/event.html
+      # libraryPath?           # might be useful if we add support for calling injectJS
+      navigationLocked: nil,   # if true, phantomjs prevents navigating away from the page. Defaults to false.
+      offlineStoragePath: nil, # file to contain offline storage data
+      offlineStorageQuota: nil, # maximum amount of data allowed in offline storage
+      ownsPages: nil,          # should child pages (opened with window.open()) be closed when parent closes?  Defaults to true.
+      paperSize: {             # the size of the rendered output http://phantomjs.org/api/webpage/property/paper-size.html
+        format: nil,           # size for pdf pages, defaults to 'A4'?
+        orientation: nil,      # orientation for pdf pages, defautls to 'portrait?'
+        width: nil,            # width of png/jpeg/gif
+        height: nil,           # height of png/jpeg/gif
+        border: nil            # blank border around the page, defaults to '1cm'?
+        # margin: nil          # use border instead
+      },
+      scrollPosition: {        # scroll page to here before rendering
+        left: nil,             # defaults to (0,0) which renders the entire page
+        top: nil
+      },
+      settings: {             # request settings: http://phantomjs.org/api/webpage/property/settings.html
+        javascriptCanCloseWindows: nil,        # whether window.open() is allowed, defaults to true
+        javascriptCanOpenWindows: nil,         # whether window.close() is allowed, defaults to true
+        javascriptEnabled: nil,                # if false, Javascript in the requested page is not executed.  Defaults to true.
+        loadImages: nil,                       # if false, inlined images in the requested page are not loaded (see also config.loadImages).  Defaults to true.
+        localToRemoteUrlAccessEnabled: nil,    # if true, local resources (like a page loaded using file:// url) are able to load remote assets.  Defaults to false.
+        password: nil,                         # password for basic HTTP authentication, see also userName
+        resourceTimeout: nil,                  # time in ms after which request will stop and onResourceTimeout() is called
+        userAgent: nil,                        # user agent string for requests (nil means use PhantomJS's default WebKitty one)
+        userName: nil,                         # name for basic HTTP authentication, see also password
+        webSecurityEnabled: nil,               # see config.webSecurity.  Defaults to true.
+        XSSAuditingEnabled: nil                # monitor requests for XSS attempts.  Defaults to false.
+      },
+      viewportSize: {            # sets the size of the virtual browser window
+        width: nil,
+        height: nil
+      },
+      zoomFactor: nil            # 4.0 increases page by 4X before rendering (right?), 0.25 shrinks page by 4X.  Defaults to 1.0.
+    }
+  }
+end