RubyGems - RedNails - Versions diffs - 1.0.0 - Mend

RedNails 1.0.0

Files changed (10) hide show

data/README ADDED

@@ -0,0 +1,76 @@
+RedNails
+Description:
+RedNails is a data scraping library that uses templates to determine what data
+to extract from actual data feeds.
+RedNails uses the template to create a regular expression that catches the
+user marker variables.  When a string of data is passed to RedNails it will
+use the regular expression to extract the matches and return them to the user.
+If the scraped data is regular enough then RedNails is a simple way to extract
+data as all one needs to do is copy a live data feed and mark the points to
+extract and make this the template.
+License: BSD
+Usage:
+1) Create a template.
+2) Load and initialize an instance of a RedNails object with the template.
+3) Pass this instance your data feed from which you wish to extract information.
+4) Use the results.
+Template Format:
+A RedNails template is simply a text file that has the points to scrape marked
+with what looks like a ruby string substitution.  You give each substitution a
+unique variable name that can be referenced when using the parse_hash method.
+An example template is:
+"Hello my name is #{name}.  How are you?"
+If the data string to scrape is:
+"Hello my name is Mr.Bill.  How are you?"
+Then the following code fragement will produce "Mr.Bill":
+ require 'rednails'
+ rednails = RedNails.new("template.txt")
+ results = rednails.parse_hash("livedata.txt")
+ puts results["name"] # => Mr.Bill
+Repetitions:
+If have data that you would like to extract which repeats itself then there is
+an additional template marker you can use.
+For the first example replace the data with #{Rep:} after the colon inside of
+the Rep marker you will then place the structured data that repeats, except
+that for each unique piece of data that you would like to extract replace it
+with a unique variable name that starts and ends with @.
+For example if you have an arbitrary list of images that you would like to
+extract you can make a template like this:
+<html>
+<body>
+A bunch of photos:
+#{Rep:<img src="@url@" alt="@txt@"/>}
+</body>
+</html>
+For more details please see the test cases.
+Installation:
+* Gem
+gem install rednails
+* Manual
+As root
+# ruby setup.rb all
+Author and Contributions:
+Zev Blut
+With some changes and help by Min Lin Hsieh, Daniel DeLorme and
+Pierre Baumard.

data/lib/rednails.rb ADDED

@@ -0,0 +1,210 @@
+#
+# Copyright (c) 2006, Ubiquitous Business Technology (http://ubit.com)
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#
+#    * Redistributions of source code must retain the above copyright
+#      notice, this list of conditions and the following disclaimer.
+#
+#    * Redistributions in binary form must reproduce the above
+#      copyright notice, this list of conditions and the following
+#      disclaimer in the documentation and/or other materials provided
+#      with the distribution.
+#
+#    * Neither the name of Ubit nor the names of its
+#      contributors may be used to endorse or promote products derived
+#      from this software without specific prior written permission.
+#
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+# == Author
+# Zev Blut (zb@ubit.com)
+require 'net/http'
+class RedNails
+  attr_reader :variables
+  # Takes a marked template file and an optional boolean that determines if an
+  # exception should be raised on errors.
+  def initialize(templatefile, raises_exception = false)
+    @templatefile = templatefile
+    @variables = nil
+    @repetitions = Hash.new
+    @regexp = parse_template(templatefile)
+    @excepts = raises_exception
+  end
+  # Takes a string of data to scrape.
+  # returns an array of variables defined in the templatefile
+  def parse(text)
+    #10 seconds timeout
+    #because those huge regular expressions can take a LONG time if there is no match
+    m = nil
+    begin
+      timeout(10) do
+        m = @regexp.match(text)
+      end
+    rescue => err
+      raise "REGEXP TIMEOUT!"
+    end
+    if m.nil?
+      if @excepts
+        raise "REGEXP from #{@templatefile} IS::::::::::::::::::::::::\n#{@regexp.source}" +
+          "COULD NOT MATCH PAGE TEXT:::::::::::::::::::::::::::::\n#{text}"
+      end
+      return nil
+    end
+    vals = []
+    # the ... means 1 to val -1 so all of the matches
+    (1...m.size).each do |i|
+      if @repetitions.key?(i)
+        reg = @repetitions[i][0]
+        vals<< m[i].scan(reg)
+      else
+        vals<< m[i]
+      end
+    end
+    return vals
+  end
+  # Takes a string of data to scrape.
+  # Returns a Hash with the template variable names as keys and matching
+  # scraped data as values.
+  def parse_hash(text)
+    vals = parse(text)
+    return nil if vals.nil?
+    hvals = {}
+    # Can probably do a block pass an yield instead of this.
+    vals.each_index do |i|
+      if @repetitions.key?(i+1)
+        varnames = @repetitions[i+1][1]
+        k=1
+        vals[i].each do |valcombo|
+          valcombo.each_index do |j|
+            hvals["#{varnames[j]}_#{k}"] = valcombo[j]
+          end
+          k+=1
+        end
+      else
+        hvals[@variables[i]] = vals[i]
+      end
+    end
+    return hvals
+  end
+  def print_detailed
+    puts "RedNails Detailed Info"
+    puts "-----------Regular Expression Source--------------"
+    puts @regexp.source
+    puts "--------------------------------------------------"
+    puts "-----------Variables------------------------------"
+    puts @variables.inspect
+    puts "--------------------------------------------------"
+    puts "-----------Repetitions------------------------------"
+    puts @repetitions.inspect
+    puts "--------------------------------------------------"
+  end
+  ###########################################################################
+  private
+  def parse_template(template)
+    templatetext = File.open(template,"r") { |f| f.read }
+    literals = Array.new
+    tail = ""
+    @variables = Array.new
+    tmptext = templatetext
+    while m = /(.*?)\#\{(.*?)\}(.*)/mi.match(tmptext)
+      literals << m[1]
+      @variables<< m[2]
+      tail = m[3]
+      tmptext = m[3]
+      reps,repnames = check_for_repetition(m[2])
+      if reps.class() == Regexp
+        @repetitions[@variables.size] = [reps,repnames]
+      end
+    end
+    # push the last matched tail onto the list
+    literals<< tail
+    literals = literals.map do |lit|
+      litexp = ""
+      # find all the whitespace items and condense to \s
+      split = lit.squeeze("\s\t\n\r\f").split(/\s/)
+      split.each_index do |i|
+        if split[i] != ""
+          litexp<< Regexp.escape(split[i])
+          # put a regexp for \s if it is not the last lit
+          # because we do not want to eat the spaces in a variable
+          if  i != (split.size - 1)
+            litexp<< "\\s*"
+          end
+        else
+          litexp<< "\\s*"
+        end
+      end
+      litexp
+    end
+    # generate the regular expression
+    regexp = "\\s*"
+    literals.each_with_index do |val,i|
+      regexp<< literals[i]
+      regexp<< "(.*)" if @variables.size > i
+    end
+    regexp<< "\\s*"
+    return Regexp.new(regexp, Regexp::MULTILINE | Regexp::IGNORECASE)
+  end
+  def check_for_repetition(var)
+    if m = /Rep:(.*)/mi.match(var)
+      varnames = Array.new
+      vals = m[1].scan(/(.*?)@(.*?)@(\S?)/mi)
+      reg = ""
+      vals.each do |lit,svar,taillit|
+        varnames<< svar
+        litexp = ""
+        # find all the whitespace items and condense to \s
+        split = lit.squeeze("\s\t\n\r\f").split(/\s/)
+        split.each_index do |i|
+          if split[i] != ""
+            litexp<< Regexp.escape(split[i])
+            # put a regexp for \s if it is not the last lit
+            # because we do not want to eat the spaces in a variable
+            if  i != (split.size - 1)
+              litexp<< "\\s*"
+            end
+          else
+            litexp<< "\\s*"
+          end
+        end
+        reg<< "#{litexp}(.*?)#{Regexp.escape(taillit)}"
+      end
+      reg<< "\\s*"
+      return Regexp.new(reg, Regexp::MULTILINE | Regexp::IGNORECASE),varnames
+    else
+      return var,nil
+    end
+  end
+end

data/test/reptemplate.html ADDED

@@ -0,0 +1,12 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html>
+  <head>
+    <title>Reps</title>
+  </head>
+  <body>
+    <h1>Reps</h1>
+    #{Rep:<img src="@url@" alt="@txt@"/>}
+    <hr>
+  </body>
+</html>

data/test/reptemplatetest.html ADDED

@@ -0,0 +1,19 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html>
+  <head>
+    <title>Reps</title>
+  </head>
+  <body>
+    <h1>Reps</h1>
+    <img src="nug1.jpg" alt="nug1"/>
+    <img src="nug2.jpg" alt="nug2"/>
+    <img src="nug3.jpg" alt="nug3"/>
+    <img src="nug4.jpg" alt="nug4"/>
+    <IMG
+      src="nug5.jpg" alt="nug5"/>
+    <hr>
+  </body>
+</html>

data/test/simpletemplate.txt ADDED

	@@ -0,0 +1,2 @@
1	+ This is a #{s} simple text!
2	+ It is #{v} simple no!

data/test/simpletest.txt ADDED

	@@ -0,0 +1,2 @@
1	+ This is a simple simple text!
2	+ It is very simple no!

data/test/tc_rednails.rb ADDED

@@ -0,0 +1,50 @@
+# $Id: tc_scrapper.rb 319 2005-12-06 05:51:40Z zb $
+#
+# Test cases for the Scrapper
+#
+# Copyright (c) 2004-2005 Ubiquitous Business Technology, Inc.
+#
+# Authors: Zev Blut
+require 'test/unit'
+require "rednails"
+RN_PATH = File.dirname(__FILE__)
+class TC_RedNails < Test::Unit::TestCase
+  def read_file(file)
+    File.open(file,"r") { |f| f.read }
+  end
+  def test_basic_template
+    stemplate = RedNails.new("#{RN_PATH}/templatefile.html")
+    test_text = read_file("#{RN_PATH}/templatefiletest.html")
+    arr = stemplate.parse(test_text)
+    assert_equal(["Header muck meader","Test Nug text","NUG2"],
+                 arr,
+                 "Extracted variables from template test file is not what was expected")
+  end
+  def test_rep_template
+    stemplate = RedNails.new("#{RN_PATH}/reptemplate.html")
+    test_text = read_file("#{RN_PATH}/reptemplatetest.html")
+    arr = stemplate.parse(test_text)
+    assert_equal([[
+                     ["nug1.jpg", "nug1"], ["nug2.jpg", "nug2"],
+                     ["nug3.jpg", "nug3"], ["nug4.jpg", "nug4"],
+                     ["nug5.jpg", "nug5"] ]],
+                 arr,
+                 "Extracted variables from template test file is not what was expected")
+    hash = stemplate.parse_hash(test_text)
+    assert_equal({ "url_1" => "nug1.jpg", "txt_1" => "nug1",
+                   "url_2" => "nug2.jpg", "txt_2" => "nug2",
+                   "url_3" => "nug3.jpg", "txt_3" => "nug3",
+                   "url_4" => "nug4.jpg", "txt_4" => "nug4",
+                   "url_5" => "nug5.jpg", "txt_5" => "nug5" },
+                 hash,
+                 "Extracted variable hash from template test file is not what was expected")
+  end
+end

data/test/templatefile.html ADDED

@@ -0,0 +1,15 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html>
+  <head>
+    <title>Stuff</title>
+  </head>
+  <Body>
+    <h1>#{header}</h1>
+    <table>
+	<tr>#{table1}</tr>
+	<tr>#{table2}</tr>
+    </table>
+    <hr>
+  </body>
+	</html>

data/test/templatefiletest.html ADDED

@@ -0,0 +1,15 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html>
+  <head>
+    <title>Stuff</title>
+  </head>
+  <body>
+    <h1>Header muck meader</h1>
+    <table>
+	<tr>Test Nug text</tr>
+	<tr>NUG2</tr>
+    </table>
+    <hr>
+  </body>
+</html>

metadata ADDED

@@ -0,0 +1,53 @@
+--- !ruby/object:Gem::Specification
+rubygems_version: 0.8.11
+specification_version: 1
+name: RedNails
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+date: 2006-12-13 00:00:00 +09:00
+summary: A template driven data scraper
+require_paths:
+- lib
+email: zb@ubit.com
+homepage: http://rednails.rubyforge.org/
+rubyforge_project: rednails
+description:
+autorequire: rednails
+default_executable:
+bindir: bin
+has_rdoc: true
+required_ruby_version: !ruby/object:Gem::Version::Requirement
+  requirements:
+  - - ">"
+    - !ruby/object:Gem::Version
+      version: 0.0.0
+  version:
+platform: ruby
+signing_key:
+cert_chain:
+authors:
+- Zev Blut
+files:
+- lib/rednails.rb
+- test/simpletemplate.txt
+- test/reptemplate.html
+- test/reptemplatetest.html
+- test/simpletest.txt
+- test/tc_rednails.rb
+- test/templatefile.html
+- test/templatefiletest.html
+- README
+test_files:
+- test/tc_rednails.rb
+rdoc_options: []
+extra_rdoc_files:
+- README
+executables: []
+extensions: []
+requirements: []
+dependencies: []