saper 0.5.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: be3fc3ae66117c4a6e4c28cb4a85834f9de5bea4
+  data.tar.gz: 41f8df60e4ee69d04868beef49a234218ec7e3d2
+SHA512:
+  metadata.gz: 6533dd86c1d9b23cf20094666131d78197a89782a5d9ff6b4237127a14ab8f6be45727461592b2cad3a1d28074b8c4abce17f2da04c95498e5213d216f00f89b
+  data.tar.gz: 8b72ff711a4d6244e0ee8655fa40cd280a51961837a73604d090c014ea3f446a6d0ecfb434ed20df3d465038e52bea9ff2151e55aacb01cdf0c101fe267847e0

data/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2013 Merimond Corporation
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,126 @@
+# Saper
+
+Saper is a web automation library written in Ruby. It allows to crawl websites and
+extract data in an efficient, controllable and fault-tolerant manner.
+
+Common use scenarios:
+
+- scrape a website and save data in a structured format
+- create an RSS feed for a website or a web application
+- create an API for a website that doesn't have one
+
+## Installing
+
+Make sure you have Ruby and RubyGems, then run:
+
+    gem install saper
+
+## Recipes
+
+Recipe is the core element of Saper. It is a chain of actions run consecutively, so
+that each action processes output of the preceding action. You may create recipes
+by instantiating Ruby classes, however using embedded DSL is the preferred method.
+
+Here's a recipe that produces a list of recent Bloomberg articles in the 'worldwide
+news' section (note that this data is unavailable via RSS):
+
+    recipe :bloomberg do
+      set_input "http://www.bloomberg.com/news/worldwide"
+      fetch
+      convert_to_html
+      find ".news_item a"
+      get_attribute "href"
+      prepend_with "http://bloomberg.com"
+    end
+    
+Given that file is saved as _myrecipe.txt_, you can now use the command line:
+
+    $ saper myrecipe.txt -recipe bloomberg
+    
+Alternatively, you can use Ruby:
+
+    #!/usr/bin/env ruby
+    Saper.run("myrecipe.txt", :bloomberg).results
+
+## Data flow
+
+Data flows from one action to another so that output of each action is used as
+input for the next one. Using the example above:
+    
+    set_input "http://www.bloomberg.com/news/worldwide"
+    > String
+    fetch
+    > Document
+    convert_to_html
+    > HTML
+    find ".news_item a"
+    > [HTML, HTML, ... ]
+    get_attribute "href"
+    > [String, String, ... ]
+    prepend_with "http://bloomberg.com"
+    > [String, String, ... ]
+    
+Whenever an action returns multiple results (e.g. *find* returns multiple HTML nodes)
+the following action will run several times as well. For instance, if *find* returns
+10 elements, then *get_attribute* will run 10 times (and produce 10 elements).
+
+If any action fails (e.g. links has no *href* attribute), Saper will silently skip it
+and proceed with the rest. All errors are logged and available for subsequent inspection,
+but no error will ever stop the execution of a recipe -- this is the core idea behind Saper.
+
+## Available actions
+
+Below is a list of all available actions:
+
+### Downloading information
+
+- **fetch** - download data from URL.
+- **convert_to_json** - parse downloaded data as JSON.
+- **convert_to_html** - parse downloaded data as HTML.
+- **convert_to_xml** - parse downloaded data as XML.
+
+### String manipulations
+
+- **convert_to_time** (format, timezone) - convert string to _time_.
+- **append_with** (string) - concatenate input with _string_.
+- **prepend_with** (string) - concatenate _string_ with input.
+- **remove_after** (separator) - search for _separator_ and remove part of input that follows the first occurrence.
+- **remove_before** (separator) - search for _separator_ and remove part of input that precedes the first occurrence.
+- **remove_matching** (regexp) - stop recipe execution for strings that don't match the specified pattern.
+- **replace** (string, string) - substitute one block of text with another.
+- **select_matching** (regexp) - continue recipe execution only for those strings that match the specified pattern.
+- **split** (separator) - split string into multiple parts using specified _separator_.
+
+### HTML / XML manipulations
+
+- **convert_to_markdown** - return tag contents converted to markdown.
+- **find** (xpath) - return nodes matching specified XPath or CSS selector.
+- **find_first** (xpath) - return the first node matching specified XPath or CSS selector.
+- **get_attribute** (name) - returns value of specified attribute.
+- **get_contents** - returns tag contents including any child tags (similar to inner_html).
+- **get_text** - returns text contents of a tag, skipping any tags (similar to inner_text).
+- **remove_tags** (name) - removes child tags including content.
+- **skip_tags** (name) - removes child tags preserving their content.
+
+### Special-purpose actions
+
+- **set_input** (string) - sets input for the following action.
+- **create_atom** - create an Atom from saved variables.
+- **run_recipe_and_save** (variable, recipe) - run another recipe and save its result as a variable.
+- **run_recipe** (recipe) - run another recipe and use it's output as input for the next action.
+- **save** (variable) - save input as a variable.
+
+# Contributing
+
+- Find something you would like to work on.
+- Fork the project and do your work in a topic branch.
+- Make sure your changes will work on both Ruby 1.8.7 and Ruby 1.9.
+- Add tests in spec/ folder for the behavior you want to test.
+- Run all the tests using _rake spec_.
+- Commit your changes and send a pull request.
+
+# License
+
+Copyright (c) 2013 Merimond Corporation. MIT license, see [LICENSE] for details.
+
+[LICENSE]: http://github.com/merimond/saper/blob/master/LICENSE

data/Rakefile

@@ -0,0 +1,17 @@
+require 'rubygems'
+require 'bundler'
+require 'rspec/core/rake_task'
+require 'yard'
+
+Bundler.require
+Bundler::GemHelper.install_tasks
+
+desc "Run specs"
+RSpec::Core::RakeTask.new do |t|
+  # nothing
+end
+
+desc "Generate docs"
+YARD::Rake::YardocTask.new do |t|
+  t.files   = ['lib/saper/**/*.rb']
+end

data/bin/saper

@@ -0,0 +1,60 @@
+#!/usr/bin/env ruby
+
+require "saper"
+require 'trollop'
+
+opts = Trollop::options do
+  banner <<-EOS
+Saper web automation library (version #{Saper::VERSION})
+Usage: saper <filename> [options]
+  
+Options are ...
+  EOS
+  opt :input, "Recipe input", :type => :string
+  opt :recipe, "Recipe name", :type => :string
+end
+
+unless ARGV.size == 1
+  Trollop::die "Please specify filename"
+end
+
+path = File.expand_path(ARGV.first)
+
+unless File.exists?(path) && File.file?(path)
+  Trollop::die("File not found")
+end
+
+begin
+  data = File.read(path)
+rescue
+  Trollop::die("File cannot be read")
+end  
+
+begin
+  space = Saper::Namespace.parse(data)
+rescue NoMethodError => e
+  Trollop::die("Invalid command: %s" % e.message.match(/`(\w+)'/)[1])
+end
+
+if opts[:recipe].nil?
+  if space.run_by_default.nil?
+    Trollop::die("Recipe name undefined")
+  end
+  opts[:recipe] = space.run_by_default
+end
+
+begin
+  recipe = space[opts[:recipe]]
+rescue Saper::RecipeNotFound => e
+  Trollop::die("`%s` recipe not found" % opts[:input])
+rescue Saper::ActionNotFound => e
+  Trollop::die("Invalid action: %s" % e.message)
+rescue Saper::InvalidArgument => e
+  Trolltop::die("Invalid action argument")
+end
+
+if opts[:input].nil? && recipe.input_required?
+  Trollop::die("No input specified, action requires %s" % recipe.input_required.inspect)
+end
+
+puts recipe.run(opts[:input]).results

data/lib/lib/json_search.rb

@@ -0,0 +1,54 @@
+# XPath-style library for JSON
+
+module JSONSearch
+  
+  def self.find(object, selector)
+    if object.is_a?(Array)
+      return object.map { |item|
+        find(item, selector)
+      }.flatten.compact
+    end
+    case selector
+    when /\A\/([^\/]+)/
+      name = $1
+      sub  = selector.sub($&, "")
+      children(object, name).map { |child|
+        find(child, sub)
+      }.flatten.compact
+    when /\A\/\/([^\/]+)/
+      name = $1
+      sub  = selector.sub($&, "")
+      descendants(object, name).map { |descendant|
+        find(descendant, sub)
+      }.flatten.compact
+    when /\A\z/
+      object
+    else
+      raise "Unsupported selector"
+    end
+  end
+  
+  def self.children(object, name)
+    case object
+    when Array
+      object.map { |item|
+        children(item, name)
+      }.flatten
+    when Hash
+      if name == "*"
+        object.values
+      else
+        [object[name]]
+      end
+    else
+      []
+    end
+  end
+  
+  def self.descendants(object, name)
+    children(object, "*").map { |child|
+      descendants(child, name)
+    }.push(*children(object, name))
+  end
+  
+end

data/lib/lib/mechanize.rb

@@ -0,0 +1,26 @@
+# Monkey patching Mechanize
+# - allow custom pre_connect_hooks and post_connect_hooks
+
+class Mechanize
+  
+  def pre_connect_hook(&block)
+    @agent.pre_connect_hook(&block)
+  end
+  
+  def post_connect_hook(&block)
+    @agent.post_connect_hook(&block)
+  end
+  
+end
+
+class Mechanize::HTTP::Agent
+  
+  def pre_connect_hook(&block)
+    @pre_connect_hooks << block
+  end
+
+  def post_connect_hook(&block)
+    @post_connect_hooks << block
+  end
+  
+end

data/lib/lib/nokogiri.rb

@@ -0,0 +1,12 @@
+# Monkey patching Nokogiri:
+# - correctly handle — character
+
+module Nokogiri
+  module HTML
+    
+    def self.parse(thing, url = nil, encoding = nil, options = XML::ParseOptions::DEFAULT_HTML, &block)
+      Document.parse(thing.gsub("—", "—"), url, encoding, options, &block)
+    end
+    
+  end
+end

data/lib/saper.rb

@@ -0,0 +1,37 @@
+module Saper
+  
+  def self.require_multiple(dir)
+    Dir[File.dirname(__FILE__) + "/" + dir].each { |file| require file }
+  end
+  
+  require 'reverse_markdown'
+  require 'json'
+  require 'vremya'
+  require 'nokogiri'
+  require 'securerandom'
+  require 'mechanize'
+
+  require_relative "lib/json_search"
+  require_relative "lib/mechanize"
+  require_relative "lib/nokogiri"
+  
+  require_relative "saper/version"
+  require_relative "saper/version"
+  require_relative "saper/core/item"
+  require_multiple "saper/items/*.rb"
+  require_relative "saper/core/argument"
+  require_multiple "saper/arguments/*.rb"
+  require_relative "saper/core/action"
+  require_multiple "saper/actions/*.rb"
+  require_relative "saper/core/error"
+  require_relative "saper/core/namespace"
+  require_relative "saper/core/type"
+  require_relative "saper/core/recipe"
+  require_relative "saper/core/error"
+  require_relative "saper/core/keychain"
+  require_relative "saper/core/dsl"
+  require_relative "saper/core/browser"
+  require_relative "saper/core/logger"
+  require_relative "saper/core/runtime"
+  
+end

data/lib/saper/actions/append_with.rb

@@ -0,0 +1,14 @@
+module Saper
+  module Actions
+    class AppendWith < Action
+      
+      argument :text
+      accepts :text, :returns => :text
+      
+      run do |input, string|
+        "%s%s" % [input, string]
+      end
+      
+    end
+  end
+end

data/lib/saper/actions/convert_to_html.rb

@@ -0,0 +1,14 @@
+module Saper
+  module Actions
+    class ConvertToHTML < Action
+      
+      accepts :text, :returns => :html
+      accepts :document, :returns => :html
+      
+      run do |input|
+        input.to_html
+      end
+      
+    end
+  end
+end

data/lib/saper/actions/convert_to_json.rb

@@ -0,0 +1,14 @@
+module Saper
+  module Actions
+    class ConvertToJSON < Action
+      
+      accepts :text, :returns => :json
+      accepts :document, :returns => :json
+      
+      run do |input|
+        input.to_json
+      end
+      
+    end
+  end
+end

data/lib/saper/actions/convert_to_markdown.rb

@@ -0,0 +1,13 @@
+module Saper
+  module Actions
+    class ConvertToMarkdown < Action
+      
+      accepts :html, :returns => :markdown
+      
+      run do |input|
+        input.to_markdown
+      end
+      
+    end
+  end
+end

data/lib/saper/actions/convert_to_time.rb

@@ -0,0 +1,15 @@
+module Saper
+  module Actions
+    class ConvertToTime < Action
+      
+      argument :text
+      argument :text, :optional => true
+      accepts :text, :returns => :time
+      
+      run do |input, format, tz|
+        input.to_time(format, tz)
+      end
+      
+    end
+  end
+end