RubyGems - RedNails - Versions diffs - 1.0.0 - Mend

RedNails 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (10) hide show

data/README ADDED

@@ -0,0 +1,76 @@
+RedNails
+Description:
+RedNails is a data scraping library that uses templates to determine what data
+to extract from actual data feeds.
+RedNails uses the template to create a regular expression that catches the
+user marker variables.  When a string of data is passed to RedNails it will
+use the regular expression to extract the matches and return them to the user.
+If the scraped data is regular enough then RedNails is a simple way to extract
+data as all one needs to do is copy a live data feed and mark the points to
+extract and make this the template.
+License: BSD
+Usage:
+1) Create a template.
+2) Load and initialize an instance of a RedNails object with the template.
+3) Pass this instance your data feed from which you wish to extract information.
+4) Use the results.
+Template Format:
+A RedNails template is simply a text file that has the points to scrape marked
+with what looks like a ruby string substitution.  You give each substitution a
+unique variable name that can be referenced when using the parse_hash method.
+An example template is:
+"Hello my name is #{name}.  How are you?"
+If the data string to scrape is:
+"Hello my name is Mr.Bill.  How are you?"
+Then the following code fragement will produce "Mr.Bill":
+ require 'rednails'
+ rednails = RedNails.new("template.txt")
+ results = rednails.parse_hash("livedata.txt")
+ puts results["name"] # => Mr.Bill
+Repetitions:
+If have data that you would like to extract which repeats itself then there is
+an additional template marker you can use.
+For the first example replace the data with #{Rep:} after the colon inside of
+the Rep marker you will then place the structured data that repeats, except
+that for each unique piece of data that you would like to extract replace it
+with a unique variable name that starts and ends with @.
+For example if you have an arbitrary list of images that you would like to
+extract you can make a template like this:
+<html>
+<body>
+A bunch of photos:
+#{Rep:<img src="@url@" alt="@txt@"/>}
+</body>
+</html>
+For more details please see the test cases.
+Installation:
+* Gem
+gem install rednails
+* Manual
+As root
+# ruby setup.rb all
+Author and Contributions:
+Zev Blut
+With some changes and help by Min Lin Hsieh, Daniel DeLorme and
+Pierre Baumard.

data/lib/rednails.rb ADDED

@@ -0,0 +1,210 @@
+#
+# Copyright (c) 2006, Ubiquitous Business Technology (http://ubit.com)
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#
+#    * Redistributions of source code must retain the above copyright
+#      notice, this list of conditions and the following disclaimer.
+#
+#    * Redistributions in binary form must reproduce the above
+#      copyright notice, this list of conditions and the following
+#      disclaimer in the documentation and/or other materials provided
+#      with the distribution.
+#
+#    * Neither the name of Ubit nor the names of its
+#      contributors may be used to endorse or promote products derived
+#      from this software without specific prior written permission.
+#
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+# == Author
+# Zev Blut (zb@ubit.com)
+require 'net/http'
+class RedNails
+  attr_reader :variables
+  # Takes a marked template file and an optional boolean that determines if an
+  # exception should be raised on errors.
+  def initialize(templatefile, raises_exception = false)
+    @templatefile = templatefile
+    @variables = nil
+    @repetitions = Hash.new
+    @regexp = parse_template(templatefile)
+    @excepts = raises_exception
+  end
+  # Takes a string of data to scrape.
+  # returns an array of variables defined in the templatefile
+  def parse(text)
+    #10 seconds timeout
+    #because those huge regular expressions can take a LONG time if there is no match
+    m = nil
+    begin
+      timeout(10) do
+        m = @regexp.match(text)
+      end
+    rescue => err
+      raise "REGEXP TIMEOUT!"
+    end
+    if m.nil?
+      if @excepts
+        raise "REGEXP from #{@templatefile} IS::::::::::::::::::::::::\n#{@regexp.source}" +
+          "COULD NOT MATCH PAGE TEXT:::::::::::::::::::::::::::::\n#{text}"
+      end
+      return nil
+    end
+    vals = []
+    # the ... means 1 to val -1 so all of the matches
+    (1...m.size).each do |i|
+      if @repetitions.key?(i)
+        reg = @repetitions[i][0]
+        vals<< m[i].scan(reg)
+      else
+        vals<< m[i]
+      end
+    end
+    return vals
+  end
+  # Takes a string of data to scrape.
+  # Returns a Hash with the template variable names as keys and matching
+  # scraped data as values.
+  def parse_hash(text)
+    vals = parse(text)
+    return nil if vals.nil?
+    hvals = {}
+    # Can probably do a block pass an yield instead of this.
+    vals.each_index do |i|
+      if @repetitions.key?(i+1)
+        varnames = @repetitions[i+1][1]
+        k=1
+        vals[i].each do |valcombo|
+          valcombo.each_index do |j|
+            hvals["#{varnames[j]}_#{k}"] = valcombo[j]
+          end
+          k+=1
+        end
+      else
+        hvals[@variables[i]] = vals[i]
+      end
+    end
+    return hvals
+  end
+  def print_detailed
+    puts "RedNails Detailed Info"
+    puts "-----------Regular Expression Source--------------"
+    puts @regexp.source
+    puts "--------------------------------------------------"
+    puts "-----------Variables------------------------------"
+    puts @variables.inspect
+    puts "--------------------------------------------------"
+    puts "-----------Repetitions------------------------------"
+    puts @repetitions.inspect
+    puts "--------------------------------------------------"
+  end
+  ###########################################################################
+  private
+  def parse_template(template)
+    templatetext = File.open(template,"r") { |f| f.read }
+    literals = Array.new
+    tail = ""
+    @variables = Array.new
+    tmptext = templatetext
+    while m = /(.*?)\#\{(.*?)\}(.*)/mi.match(tmptext)
+      literals << m[1]
+      @variables<< m[2]
+      tail = m[3]
+      tmptext = m[3]
+      reps,repnames = check_for_repetition(m[2])
+      if reps.class() == Regexp
+        @repetitions[@variables.size] = [reps,repnames]
+      end
+    end
+    # push the last matched tail onto the list
+    literals<< tail
+    literals = literals.map do |lit|
+      litexp = ""
+      # find all the whitespace items and condense to \s
+      split = lit.squeeze("\s\t\n\r\f").split(/\s/)
+      split.each_index do |i|
+        if split[i] != ""
+          litexp<< Regexp.escape(split[i])
+          # put a regexp for \s if it is not the last lit
+          # because we do not want to eat the spaces in a variable
+          if  i != (split.size - 1)
+            litexp<< "\\s*"
+          end
+        else
+          litexp<< "\\s*"
+        end
+      end
+      litexp
+    end
+    # generate the regular expression
+    regexp = "\\s*"
+    literals.each_with_index do |val,i|
+      regexp<< literals[i]
+      regexp<< "(.*)" if @variables.size > i
+    end
+    regexp<< "\\s*"
+    return Regexp.new(regexp, Regexp::MULTILINE | Regexp::IGNORECASE)
+  end
+  def check_for_repetition(var)
+    if m = /Rep:(.*)/mi.match(var)
+      varnames = Array.new
+      vals = m[1].scan(/(.*?)@(.*?)@(\S?)/mi)
+      reg = ""
+      vals.each do |lit,svar,taillit|
+        varnames<< svar
+        litexp = ""
+        # find all the whitespace items and condense to \s
+        split = lit.squeeze("\s\t\n\r\f").split(/\s/)
+        split.each_index do |i|
+          if split[i] != ""
+            litexp<< Regexp.escape(split[i])
+            # put a regexp for \s if it is not the last lit
+            # because we do not want to eat the spaces in a variable
+            if  i != (split.size - 1)
+              litexp<< "\\s*"
+            end
+          else
+            litexp<< "\\s*"
+          end
+        end
+        reg<< "#{litexp}(.*?)#{Regexp.escape(taillit)}"
+      end
+      reg<< "\\s*"
+      return Regexp.new(reg, Regexp::MULTILINE | Regexp::IGNORECASE),varnames
+    else
+      return var,nil
+    end
+  end
+end

data/test/reptemplate.html ADDED

@@ -0,0 +1,12 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html>
+  <head>
+    <title>Reps</title>
+  </head>
+  <body>
+    <h1>Reps</h1>
+    #{Rep:<img src="@url@" alt="@txt@"/>}
+    <hr>
+  </body>
+</html>

data/test/reptemplatetest.html ADDED

@@ -0,0 +1,19 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html>
+  <head>
+    <title>Reps</title>
+  </head>
+  <body>
+    <h1>Reps</h1>
+    <img src="nug1.jpg" alt="nug1"/>
+    <img src="nug2.jpg" alt="nug2"/>
+    <img src="nug3.jpg" alt="nug3"/>
+    <img src="nug4.jpg" alt="nug4"/>
+    <IMG
+      src="nug5.jpg" alt="nug5"/>
+    <hr>
+  </body>
+</html>

data/test/simpletemplate.txt ADDED

	@@ -0,0 +1,2 @@
1	+ This is a #{s} simple text!
2	+ It is #{v} simple no!

data/test/simpletest.txt ADDED

	@@ -0,0 +1,2 @@
1	+ This is a simple simple text!
2	+ It is very simple no!

data/test/tc_rednails.rb ADDED

@@ -0,0 +1,50 @@
+# $Id: tc_scrapper.rb 319 2005-12-06 05:51:40Z zb $
+#
+# Test cases for the Scrapper
+#
+# Copyright (c) 2004-2005 Ubiquitous Business Technology, Inc.
+#
+# Authors: Zev Blut
+require 'test/unit'
+require "rednails"
+RN_PATH = File.dirname(__FILE__)
+class TC_RedNails < Test::Unit::TestCase
+  def read_file(file)
+    File.open(file,"r") { |f| f.read }
+  end
+  def test_basic_template
+    stemplate = RedNails.new("#{RN_PATH}/templatefile.html")
+    test_text = read_file("#{RN_PATH}/templatefiletest.html")
+    arr = stemplate.parse(test_text)
+    assert_equal(["Header muck meader","Test Nug text","NUG2"],
+                 arr,
+                 "Extracted variables from template test file is not what was expected")
+  end
+  def test_rep_template
+    stemplate = RedNails.new("#{RN_PATH}/reptemplate.html")
+    test_text = read_file("#{RN_PATH}/reptemplatetest.html")
+    arr = stemplate.parse(test_text)
+    assert_equal([[
+                     ["nug1.jpg", "nug1"], ["nug2.jpg", "nug2"],
+                     ["nug3.jpg", "nug3"], ["nug4.jpg", "nug4"],
+                     ["nug5.jpg", "nug5"] ]],
+                 arr,
+                 "Extracted variables from template test file is not what was expected")
+    hash = stemplate.parse_hash(test_text)
+    assert_equal({ "url_1" => "nug1.jpg", "txt_1" => "nug1",
+                   "url_2" => "nug2.jpg", "txt_2" => "nug2",
+                   "url_3" => "nug3.jpg", "txt_3" => "nug3",
+                   "url_4" => "nug4.jpg", "txt_4" => "nug4",
+                   "url_5" => "nug5.jpg", "txt_5" => "nug5" },
+                 hash,
+                 "Extracted variable hash from template test file is not what was expected")
+  end
+end

data/test/templatefile.html ADDED

@@ -0,0 +1,15 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html>
+  <head>
+    <title>Stuff</title>
+  </head>
+  <Body>
+    <h1>#{header}</h1>
+    <table>
+	<tr>#{table1}</tr>
+	<tr>#{table2}</tr>
+    </table>
+    <hr>
+  </body>
+	</html>

data/test/templatefiletest.html ADDED

@@ -0,0 +1,15 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html>
+  <head>
+    <title>Stuff</title>
+  </head>
+  <body>
+    <h1>Header muck meader</h1>
+    <table>
+	<tr>Test Nug text</tr>
+	<tr>NUG2</tr>
+    </table>
+    <hr>
+  </body>
+</html>

metadata ADDED

@@ -0,0 +1,53 @@
+--- !ruby/object:Gem::Specification
+rubygems_version: 0.8.11
+specification_version: 1
+name: RedNails
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+date: 2006-12-13 00:00:00 +09:00
+summary: A template driven data scraper
+require_paths:
+- lib
+email: zb@ubit.com
+homepage: http://rednails.rubyforge.org/
+rubyforge_project: rednails
+description:
+autorequire: rednails
+default_executable:
+bindir: bin
+has_rdoc: true
+required_ruby_version: !ruby/object:Gem::Version::Requirement
+  requirements:
+  - - ">"
+    - !ruby/object:Gem::Version
+      version: 0.0.0
+  version:
+platform: ruby
+signing_key:
+cert_chain:
+authors:
+- Zev Blut
+files:
+- lib/rednails.rb
+- test/simpletemplate.txt
+- test/reptemplate.html
+- test/reptemplatetest.html
+- test/simpletest.txt
+- test/tc_rednails.rb
+- test/templatefile.html
+- test/templatefiletest.html
+- README
+test_files:
+- test/tc_rednails.rb
+rdoc_options: []
+extra_rdoc_files:
+- README
+executables: []
+extensions: []
+requirements: []
+dependencies: []