ariel 0.0.1 → 0.1.0

data/README

@@ -1,98 +1,64 @@
-= Ariel release 0.0.1
+= Ariel release 0.1.0
+
+== About - Ariel: A Ruby Information Extraction Library
+Ariel is a library that allows you to extract information from semi-structured
+documents (such as websites). It is different to existing tools because rather
+than expecting the developer to write rules to extract the desired information,
+Ariel will use a small number of labeled examples to generate and learn
+effective extraction rules. It is developed by Alex Bradbury and released under
+the MIT license. Ariel was started as a Google Summer of Code project mentored
+by Austin Ziegler in 2006.
 
 == Install
 gem install ariel
 
 == Announcement
-This is the first public release of Ariel - A Ruby Information Extraction
-Library. See my previous post, ruby-talk:200140[http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/200140]
-for more background information. This release supports defining a tree document
-structure and learning rules to extract each node of this true. Handling of list
-extraction and learning is not yet implemented, and is the next immediate
-priority. See the examples directory included in this release and below for
-discussion of the included examples. Rule learning is functional, and appears to
-work well, but many refinements are possible. Look out for more updates and a
-new releases shortly.
-
-== About Ariel
-Ariel intends to assist in extracting information from semi-structured
-documents including (but not in any way limited to) web pages. Although you
-may use libraries such as Hpricot or Rubyful Soup, or even plain Regular
-Expressions to achieve the same goal, Ariel approaches the problem very
-differently. Ariel relies on the user labeling examples of the data they
-want to extract, and then finds patterns across several such labeled
-examples in order to produce a set of general rules for extracting this
-information from any similar document. It uses the MIT license.
-
-== Examples
-This release includes two examples in the example directory (which should now
-be in the directory to which rubygems installed ariel). The first is the
-google_calculator directory (inspired by Justin Bailey's post to my Ariel
-progress report). The structure is very simple, a calculation is extracted from
-the page, and then the actual result is extracted from that calculation. 3
-labeled examples are included. Ariel reads each of these, tokenizes them,
-and extracts each label. 4 sets of rules are learnt:
-1. Rules to locate the start of the calculation in the original document.
-2. Rules to locate the end of the calculation in the original document (applied
-   from the end of the document).
-3. Rules to locate the start of the result of the calculation from the
-   extracted calculation.
-4. Rules to locate the end of the result of the calculation from the extracted
-   calculation (applied from the end of the calculation).
-
-Take note of 3 and 4 - this is the advantage of treating a document as a tree in
-this way. Deeply nested elements can be located by generating a series of simple
-rules, rather than generating a rule with complexity that increases at each
-level. Sets of rules are generated because it may not be possible to generate a
-single rule that will catch all cases. A rule is found that matches as many of
-the examples as possible (and fails on the rest), these examples are then removed
-and a rule is found that will match as many of the remaining examples and so on.
-When it comes to applying these learnt rules, the rules are applied in order
-until there is a rule that matches.
-
-To see this example for yourself just execute structure.rb in the
-examples/google_calculator directory to create a locally writable
-structure.yaml. Then do:
-  ariel -D -m learn -s structure.yaml -d /path/to/examples/google_calculator/labeled
-
-You'll have to wait a while (see my note about performance below). At the end,
-the learnt rules will be printed in YAML format, and structure.yaml will be
-updated to include these rules. Apply these learnt rules to some unlabeled
-documents by doing:
-  ariel -D -m extract -s structure.yaml -d /path/to/examples/google_calculator/unlabeled
-
-You should see the results of a successful extraction printed to your terminal,
-such as this one:
 
-  Results for unlabeled/2:
-  calculation: 3.5 U.S. dollars = 1.8486241 British pounds
-  result: 1.8486241 British pounds
+I'm happy to announce the release of Ariel 0.1.0, the result of my Summer of
+Code work. This release should be easy to use, very functional, and hopefully
+useful - so it's worth trying out. I've put a lot of effort in to writing clear
+and straightforward documentation to get your started, so take a look at the
+docs available at http://ariel.rubyforge.org. In particular, flick through the
+tutorial and quick start guide. If you're interested, you may also want to take
+a look at the theory page where I've made a good start on describing the method
+Ariel uses to learn extraction rules. If you have any problems or find any bugs,
+just send me an email or add it to the issue tracker (see link below). Enjoy.
+See the FAQ for a vim snippet to make labeling examples a little easier.
 
-The second example (raa) learns rules using just 2 labeled examples. This is probably
-fewer than I'd recommend in most cases, but as it works... This example consists
-of project entries in the Ruby Application Archive. The structure of the page is
-very flat, so all rules are applied to the full page. Rules are learnt and
-applied as shown above. The structure.yaml files included in the examples
-directories already include rules generated by Ariel, use these if you just want
-to see extraction working.
+== Quickstart/Basic usage
 
-Note: The interface demonstrated by ariel above is not very flexible or
-friendly, it's just to serve as a demonstration for the moment.
+* @require 'ariel'@
+* Define a structure for the information you wish to extract: 
+    structure = Ariel::Node::Structure.new do |r|
+      r.item :title
+      r.item :body
+      r.list :comments do |c|
+        c.list_item :comment do |d|
+          d.item :author
+          d.item :body
+        end
+      end
+     end
+* Collect a few examples of the sort of document you wish to extract information
+  from (pages from the same website for instance).
+* Label each example with tags such as <l:title>, <l:comment> and so on in the
+  relevant places.
+*  Ariel.learn structure, labeled_file1, labeled_file2, labeled_file3
+* Find the documents you want to extract information from.
+*  extractions = Ariel.extract structure, unlabeled_file1,
+  unlabeled_file2
+*  extractions[0].search('comments/*/body').each {|e| puts e.extracted_text} =>
+  "Great stuff, loving it", "I love life", .....
+*  extractions[0].at('comments/34') => nil</tt> (there is no 34th comment, #at
+  returns the first result rather than an array of matches).
 
-== Performance
-Generating rules takes quite a long time. It is always going to be an intensive
-operation, but there are some very simple and obvious improvements in efficiency
-that can be made. For a start, the rule candidate refining process currently
-re-applies the same rules over and over every time the remaining rule candidates
-are ranked. This is where most time is spent, and caching these should make a
-big difference. This will definitely be implemented. Other performance
-enhancements are bound to be there, but my focus at this time is to get
-something that works.
 
 == Credits
 Ariel is developed by Alex Bradbury as a Google Summer of Code project under the
 mentoring of Austin Ziegler.
 
 == Links
-Watch my development through the subversion repository at http://rubyforge.org/projects/ariel
-I've also just started using the tracker at http://code.google.com/p/ariel/
+SVN Repository: http://rubyforge.org/projects/ariel
+Issue tracker: http://code.google.com/p/ariel/issues/
+Documentation/homepage: http://ariel.rubyforge.org
+RDoc: http://ariel.rubyforge.org/rdoc/

data/bin/ariel

@@ -14,43 +14,52 @@ OptionParser.new do |opts|
   end
 
   opts.on('-d', '--dir=DIRECTORY', 'Directory to look for documents to operate on.') do |dir|
+    raise ArgumentError, "directory does not exist" unless FileTest.directory? dir
     options[:dir]=dir
   end
 
-  opts.on('-D', '--debug', 'Directory to look for documents to operate on.') do 
+  opts.on('-D', '--debug', 'Enable debugging output.') do 
     $DEBUG=true
   end
 
   opts.on('-s', '--structure=STRUCTURE', 'YAML file in which the structure is defined') do |structure|
     options[:structure]=structure
   end
+
+  opts.on('-o', '--output-dir=DIRECTORY', 'Directory to output to') do |dir|
+    raise ArgumentError, "directory does not exist" unless FileTest.directory? dir
+    options[:output_dir]=dir
+  end
 end.parse!
 
-require 'ariel' #After option parsing to debug setting can take effect
+require 'ariel' #After option parsing so debug setting can take effect
+
+files=Dir["#{options[:dir]}/*"].select {|file_name| File.file? file_name}
+structure=YAML.load_file options[:structure]
 
 case options[:mode]
 when "learn"
-  structure=YAML.load_file options[:structure]
-  learnt_structure=Ariel::ExampleDocumentLoader.load_directory options[:dir], structure
+  Ariel.learn(structure, *files)
   File.open(options[:structure], 'wb') do |file|
-    YAML.dump(learnt_structure, file)
-  end
-  learnt_structure.each_descendant do |structure_node|
-    puts structure_node.meta.name.to_s
-    puts structure_node.ruleset.to_yaml
+    YAML.dump(structure, file)
   end
+
 when "extract"
-  learnt_structure=YAML.load_file options[:structure]
-  Dir.glob("#{options[:dir]}/*") do |file|
-    tokenstream=Ariel::TokenStream.new
-    tokenstream.tokenize File.read(file)
-    root_node=Ariel::ExtractedNode.new :root, tokenstream, learnt_structure
-    learnt_structure.apply_extraction_tree_on root_node
-    puts "Results for #{file}:"
-    root_node.each_descendant do |node|
-      puts "#{node.meta.name}: #{node.tokenstream.text}"
+  extractions = Ariel.extract(structure, *files)
+  if options[:output_dir]
+    extractions.zip(files) do |extraction, file|
+      filename=File.join(options[:output_dir], File.basename(file)+'.yaml')
+      File.open(filename, 'wb') do |f|
+        YAML.dump(extraction, f)
+      end
+    end
+  else
+    puts "No --output-dir given, so printing extractions to stdout"
+    extractions.each do |extraction| 
+      extraction.each_descendant do |node|
+        puts "#{node.node_name}: #{node.tokenstream.text}"
+      end
+      puts #Blank line looks prettier
     end
-    puts
-   # puts root_node.to_yaml
   end
 end

data/examples/google_calculator/structure.rb

@@ -1,12 +1,12 @@
 require 'ariel'
 require 'yaml'
 
-structure = Ariel::StructureNode.new do |r|
+structure = Ariel::Node::Structure.new do |r|
   r.item :calculation do |c|
     c.item :result
   end
 end
 
-File.open('structure.yaml') do |file|
+File.open('structure.yaml', 'w') do |file|
   YAML.dump structure, file
 end

data/examples/google_calculator/structure.yaml

@@ -1,46 +1,44 @@
---- &id002 !ruby/object:Ariel::StructureNode 
+--- &id002 !ruby/object:Ariel::Node::Structure 
 children: 
-  :calculation: &id001 !ruby/object:Ariel::StructureNode 
+  :calculation: &id001 !ruby/object:Ariel::Node::Structure 
     children: 
-      :result: !ruby/object:Ariel::StructureNode 
+      :result: !ruby/object:Ariel::Node::Structure 
         children: {}
 
-        meta: !ruby/object:OpenStruct 
-          table: 
-            :node_type: :not_list
-            :name: :result
+        node_name: :result
+        node_type: :not_list
         parent: *id001
         ruleset: !ruby/object:Ariel::RuleSet 
           end_rules: 
           - !ruby/object:Ariel::Rule 
             direction: :back
+            exhaustive: false
             landmarks: []
 
           start_rules: 
           - !ruby/object:Ariel::Rule 
             direction: :forward
+            exhaustive: false
             landmarks: 
             - - "="
-    meta: !ruby/object:OpenStruct 
-      table: 
-        :node_type: :not_list
-        :name: :calculation
+    node_name: :calculation
+    node_type: :not_list
     parent: *id002
     ruleset: !ruby/object:Ariel::RuleSet 
       end_rules: 
       - !ruby/object:Ariel::Rule 
         direction: :back
+        exhaustive: false
         landmarks: 
         - - </b>
         - - </b>
       start_rules: 
       - !ruby/object:Ariel::Rule 
         direction: :forward
+        exhaustive: false
         landmarks: 
         - - <b>
         - - gif
         - - <b>
-meta: !ruby/object:OpenStruct 
-  table: 
-    :node_type: :not_list
-    :name: :root
+node_name: :root
+node_type: :not_list

data/examples/raa/labeled/highline.html

@@ -96,17 +96,18 @@ highline / <l:current_version>1.2.0</l:current_version>
 
   <tr><th>Versions: </th>
     <td>
-      <l:version_history>[<a href="project/highline/1.2.0">1.2.0</a> (2006-03-23)]
+      <l:version_history>[<a
+      href="project/highline/1.2.0"><l:version>1.2.0</l:version></a> (2006-03-23)]
 
         [<a href="project/highline/1.0.2">1.0.2</a> (2006-02-20)]
 
-        [<a href="project/highline/1.0.1">1.0.1</a> (2005-07-07)]
+        [<a href="project/highline/1.0.1"><l:version>1.0.1</l:version></a> (2005-07-07)]
 
         [<a href="project/highline/1.0.0">1.0.0</a> (2005-07-07)]
 
-        [<a href="project/highline/0.6.1">0.6.1</a> (2005-05-26)]
+        [<a href="project/highline/0.6.1"><l:version>0.6.1</l:version></a> (2005-05-26)]
 
-        [<a href="project/highline/0.6.0">0.6.0</a>
+        [<a href="project/highline/0.6.0"><l:version>0.6.0</l:version></a>
         (2005-05-21)]</l:version_history>
 
     </td>

data/examples/raa/labeled/mongrel.html

@@ -126,21 +126,22 @@ mongrel / <l:current_version>0.3.12</l:current_version>
 
   <tr><th>Versions: </th>
     <td>
-      <l:version_history>[<a href="project/mongrel/0.3.12">0.3.12</a> (2006-03-30)]
+      <l:version_history>[<a
+href="project/mongrel/0.3.12"><l:version>0.3.12</l:version></a> (2006-03-30)]
 
-        [<a href="project/mongrel/0.3.11">0.3.11</a> (2006-03-15)]
+        [<a href="project/mongrel/0.3.11"><l:version>0.3.11</l:version></a> (2006-03-15)]
 
-        [<a href="project/mongrel/0.3.10">0.3.10</a> (2006-03-12)]
+        [<a href="project/mongrel/0.3.10"><l:version>0.3.10</l:version></a> (2006-03-12)]
 
-        [<a href="project/mongrel/0.3.9">0.3.9</a> (2006-03-06)]
+        [<a href="project/mongrel/0.3.9"><l:version>0.3.9</l:version></a> (2006-03-06)]
 
-        [<a href="project/mongrel/0.3.8">0.3.8</a> (2006-03-04)]
+        [<a href="project/mongrel/0.3.8"><l:version>0.3.8</l:version></a> (2006-03-04)]
 
-        [<a href="project/mongrel/0.3.6">0.3.6</a> (2006-02-23)]
+        [<a href="project/mongrel/0.3.6"><l:version>0.3.6</l:version></a> (2006-02-23)]
 
-        [<a href="project/mongrel/0.3.2">0.3.2</a> (2006-02-13)]
+        [<a href="project/mongrel/0.3.2"><l:version>0.3.2</l:version></a> (2006-02-13)]
 
-        [<a href="project/mongrel/0.3.1">0.3.1</a> (2006-02-12)]</l:version_history>
+        [<a href="project/mongrel/0.3.1"><l:version>0.3.1</l:version></a> (2006-02-12)]</l:version_history>
 
     </td>
   </tr>

data/examples/raa/structure.rb

@@ -1,7 +1,7 @@
 require 'ariel'
 require 'yaml'
 
-structure = Ariel::StructureNode.new do |r|
+structure = Ariel::Node::Structure.new do |r|
   r.item :name
   r.item :current_version
   r.item :short_description
@@ -9,7 +9,9 @@ structure = Ariel::StructureNode.new do |r|
   r.item :owner
   r.item :homepage
   r.item :license
-  r.item :version_history
+  r.list :version_history do |v|
+    v.list_item :version
+  end
 end
 
 File.open('structure.yaml', 'wb') do |file|

data/examples/raa/structure.yaml

@@ -1,38 +1,16 @@
---- &id001 !ruby/object:Ariel::StructureNode 
+--- &id001 !ruby/object:Ariel::Node::Structure 
 children: 
-  :version_history: !ruby/object:Ariel::StructureNode 
+  :short_description: !ruby/object:Ariel::Node::Structure 
     children: {}
 
-    meta: !ruby/object:OpenStruct 
-      table: 
-        :name: :version_history
-        :node_type: :not_list
-    parent: *id001
-    ruleset: !ruby/object:Ariel::RuleSet 
-      end_rules: 
-      - !ruby/object:Ariel::Rule 
-        direction: :back
-        landmarks: 
-        - - </td>
-      start_rules: 
-      - !ruby/object:Ariel::Rule 
-        direction: :forward
-        landmarks: 
-        - - <td>
-        - - Versions
-        - - <td>
-  :short_description: !ruby/object:Ariel::StructureNode 
-    children: {}
-
-    meta: !ruby/object:OpenStruct 
-      table: 
-        :name: :short_description
-        :node_type: :not_list
+    node_name: :short_description
+    node_type: :not_list
     parent: *id001
     ruleset: !ruby/object:Ariel::RuleSet 
       end_rules: 
       - !ruby/object:Ariel::Rule 
         direction: :back
+        exhaustive: false
         landmarks: 
         - - </td>
         - - Category
@@ -40,109 +18,109 @@ children:
       start_rules: 
       - !ruby/object:Ariel::Rule 
         direction: :forward
+        exhaustive: false
         landmarks: 
         - - <td>
-  :current_version: !ruby/object:Ariel::StructureNode 
+  :homepage: !ruby/object:Ariel::Node::Structure 
     children: {}
 
-    meta: !ruby/object:OpenStruct 
-      table: 
-        :name: :current_version
-        :node_type: :not_list
+    node_name: :homepage
+    node_type: :not_list
     parent: *id001
     ruleset: !ruby/object:Ariel::RuleSet 
       end_rules: 
       - !ruby/object:Ariel::Rule 
         direction: :back
+        exhaustive: false
         landmarks: 
-        - - </p>
-        - - table
-        - - </p>
+        - - </a>
+        - - Download
+        - - </a>
       start_rules: 
       - !ruby/object:Ariel::Rule 
         direction: :forward
+        exhaustive: false
         landmarks: 
-        - - /
-        - - caption
-        - - /
-  :homepage: !ruby/object:Ariel::StructureNode 
+        - - ">"
+        - - rubyforge
+        - - ">"
+  :category: !ruby/object:Ariel::Node::Structure 
     children: {}
 
-    meta: !ruby/object:OpenStruct 
-      table: 
-        :name: :homepage
-        :node_type: :not_list
+    node_name: :category
+    node_type: :not_list
     parent: *id001
     ruleset: !ruby/object:Ariel::RuleSet 
       end_rules: 
       - !ruby/object:Ariel::Rule 
         direction: :back
+        exhaustive: false
         landmarks: 
-        - - </a>
-        - - Download
-        - - </a>
+        - - </td>
+        - - Status
+        - - </td>
       start_rules: 
       - !ruby/object:Ariel::Rule 
         direction: :forward
+        exhaustive: false
         landmarks: 
-        - - ">"
-        - - rubyforge
-        - - ">"
-  :category: !ruby/object:Ariel::StructureNode 
+        - - <td>
+        - - <td>
+  :current_version: !ruby/object:Ariel::Node::Structure 
     children: {}
 
-    meta: !ruby/object:OpenStruct 
-      table: 
-        :name: :category
-        :node_type: :not_list
+    node_name: :current_version
+    node_type: :not_list
     parent: *id001
     ruleset: !ruby/object:Ariel::RuleSet 
       end_rules: 
       - !ruby/object:Ariel::Rule 
         direction: :back
+        exhaustive: false
         landmarks: 
-        - - </td>
-        - - Status
-        - - </td>
+        - - </p>
+        - - table
+        - - </p>
       start_rules: 
       - !ruby/object:Ariel::Rule 
         direction: :forward
+        exhaustive: false
         landmarks: 
-        - - <td>
-        - - <td>
-  :name: !ruby/object:Ariel::StructureNode 
+        - - :anything
+        - - caption
+        - - /
+  :name: !ruby/object:Ariel::Node::Structure 
     children: {}
 
-    meta: !ruby/object:OpenStruct 
-      table: 
-        :name: :name
-        :node_type: :not_list
+    node_name: :name
+    node_type: :not_list
     parent: *id001
     ruleset: !ruby/object:Ariel::RuleSet 
       end_rules: 
       - !ruby/object:Ariel::Rule 
         direction: :back
+        exhaustive: false
         landmarks: 
         - - </title>
       start_rules: 
       - !ruby/object:Ariel::Rule 
         direction: :forward
+        exhaustive: false
         landmarks: 
         - - "-"
         - - RAA
           - "-"
-  :owner: !ruby/object:Ariel::StructureNode 
+  :owner: !ruby/object:Ariel::Node::Structure 
     children: {}
 
-    meta: !ruby/object:OpenStruct 
-      table: 
-        :name: :owner
-        :node_type: :not_list
+    node_name: :owner
+    node_type: :not_list
     parent: *id001
     ruleset: !ruby/object:Ariel::RuleSet 
       end_rules: 
       - !ruby/object:Ariel::Rule 
         direction: :back
+        exhaustive: false
         landmarks: 
         - - </a>
         - - id
@@ -150,22 +128,22 @@ children:
       start_rules: 
       - !ruby/object:Ariel::Rule 
         direction: :forward
+        exhaustive: false
         landmarks: 
         - - ">"
         - - Owner
         - - ">"
-  :license: !ruby/object:Ariel::StructureNode 
+  :license: !ruby/object:Ariel::Node::Structure 
     children: {}
 
-    meta: !ruby/object:OpenStruct 
-      table: 
-        :name: :license
-        :node_type: :not_list
+    node_name: :license
+    node_type: :not_list
     parent: *id001
     ruleset: !ruby/object:Ariel::RuleSet 
       end_rules: 
       - !ruby/object:Ariel::Rule 
         direction: :back
+        exhaustive: false
         landmarks: 
         - - </td>
         - - Dependency
@@ -173,11 +151,49 @@ children:
       start_rules: 
       - !ruby/object:Ariel::Rule 
         direction: :forward
+        exhaustive: false
         landmarks: 
         - - <td>
         - - License
         - - <td>
-meta: !ruby/object:OpenStruct 
-  table: 
-    :name: :root
-    :node_type: :not_list
+  :version_history: &id002 !ruby/object:Ariel::Node::Structure 
+    children: 
+      :version: !ruby/object:Ariel::Node::Structure 
+        children: {}
+
+        node_name: :version
+        node_type: :list_item
+        parent: *id002
+        ruleset: !ruby/object:Ariel::RuleSet 
+          end_rules: 
+          - !ruby/object:Ariel::Rule 
+            direction: :back
+            exhaustive: true
+            landmarks: 
+            - - </a>
+          start_rules: 
+          - !ruby/object:Ariel::Rule 
+            direction: :forward
+            exhaustive: true
+            landmarks: 
+            - - ">"
+    node_name: :version_history
+    node_type: :not_list
+    parent: *id001
+    ruleset: !ruby/object:Ariel::RuleSet 
+      end_rules: 
+      - !ruby/object:Ariel::Rule 
+        direction: :back
+        exhaustive: false
+        landmarks: 
+        - - </td>
+      start_rules: 
+      - !ruby/object:Ariel::Rule 
+        direction: :forward
+        exhaustive: false
+        landmarks: 
+        - - <td>
+        - - Versions
+        - - <td>
+node_name: :root
+node_type: :not_list