titi 0.0.2 → 0.0.3

data/LICENSE.textile

@@ -0,0 +1,96 @@
+The titi code is __Copyright (c) 2010 Infochimps, Inc__
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at "http://www.apache.org/licenses/LICENSE-2.0":http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an **AS IS** BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
+
+h1. Apache License
+
+                              Apache License
+                        Version 2.0, January 2004
+                     http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+<notextile><div class="toggle"></notextile>
+
+h2. 1. Definitions.
+
+* **License** shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
+* **Licensor** shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
+* **Legal Entity** shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, **control** means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
+* **You** (or **Your**) shall mean an individual or Legal Entity exercising permissions granted by this License.
+* **Source** form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
+* **Object** form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
+* **Work** shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
+* **Derivative Works** shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
+* **Contribution** shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, **submitted** means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
+* **Contributor** shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. 2. Grant of Copyright License.
+
+Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
+
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. 3. Grant of Patent License.
+
+Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. 4. Redistribution.
+
+You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
+
+# You must give any other recipients of the Work or Derivative Works a copy of this License; and
+# You must cause any modified files to carry prominent notices stating that You changed the files; and
+# You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
+# If the Work includes a __NOTICE__ text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
+
+You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. 5. Submission of Contributions.
+
+Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. 6. Trademarks.
+
+This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. 7. Disclaimer of Warranty.
+
+Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an **AS IS** BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. 8. Limitation of Liability.
+
+In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. 9. Accepting Warranty or Additional Liability.
+
+While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+<notextile></div><div class="toggle"></notextile>
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an **AS IS** BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
+
+<notextile></div></notextile>

data/README.textile

@@ -9,8 +9,31 @@ h3. BUGS:
 
 h3. Example
 
-  <pre><code>
+Let's define a super-simple twitter user and status model:
 
+<pre><code>
+    User = Struct.new(
+      :id, :screen_name, :protected, :followers_count, :friends_count, :statuses_count, :favourites_count, :created_at,
+      :name, :url, :location, :description, :time_zone, :utc_offset,
+      :profile_background_color, :profile_text_color, :profile_link_color, :profile_sidebar_border_color, :profile_sidebar_fill_color, :profile_background_tile, :profile_background_image_url, :profile_image_url
+      )
+
+    Status = Struct.new( :id, :created_at, :user, :favorited, :truncated,
+      :in_reply_to_user_id, :in_reply_to_status_id, :text, :source,
+      :in_reply_to_screen_name )
+</code></pre>
+
+With no other work, we can pull a raw JSON hash off the wire and create model instances:
+
+<pre><code>
+    raw_json_resp = RestClient.get("http://twitter.com/statuses/show/12233609555.json")
+    Twitter::Status.adapt(JSON.load(raw_json_resp.to_s))
+    #=> #<struct Titi::Provider::Twitter::Status id=12233609555, created_at="Thu Apr 15 17:01:52 +0000 2010", user=#<struct Titi::Provider::Twitter::User id=1468401, screen_name="sockington", protected=false, followers_count=1520814, friends_count=457, statuses_count=6234, favourites_count=2, created_at=Mon, 19 Mar 2007 03:45:00 +0000, name="Sockamillion", url="http://www.sockington.org/", location="Waltham, MA", description="I am Jason Scott's Cat.", time_zone="Eastern Time (US & Canada)", utc_offset=-18000, profile_background_color="48484c", profile_text_color="000000", profile_link_color="000000", profile_sidebar_border_color="79c021", profile_sidebar_fill_color="585e7e", profile_background_tile=false, profile_background_image_url="http://a1.twimg.com/profile_background_images/6682718/SocksTwitter.jpg", profile_image_url="http://a3.twimg.com/profile_images/77537329/IMG_3738_normal.JPG">, favorited=false, truncated=false, in_reply_to_user_id=nil, in_reply_to_status_id=nil, text="THANK GOODNESS THE LIBRARY OF CONGRESS HAS UNDERSTOOD THE IMPORTANCE OF MY TWEETS what do you mean others are getting in too", source="<a href=\"http://apiwiki.twitter.com/\" rel=\"nofollow\">API</a>", in_reply_to_screen_name=nil>
+</code></pre>
+
+Titi lets you write a straightforward and compact adaptor to activity stream format:
+
+<pre><code>
     class Titi::Provider::Twitter::Status
       def to_activity_stream_entry
         ActivityStreams::Entry.adapt(
@@ -30,12 +53,19 @@ h3. Example
           end
         end
       end
-  
+    end
+</code></pre>
+
+
+Where the mapping is direct, you can use an attribute => attribute hash, as shown in the call to ActivityStreams::Entry.adapt. For anything more complex, you can run arbitrary code in the method's block _(the part within the do ... end segment)_
+
+Here's example output. _note: this isn't canonical activity streams format: the XML serialization needs work._
+
+<pre><code>
     tweet = Twitter::Status.get(12233609555)
     entry = tweet.to_activity_stream_entry
-
     puts entry.to_xml
-  
+    
         <?xml version="1.0" encoding="UTF-8"?>
         <entry>
           <verb type="symbol">post</verb>
@@ -71,24 +101,8 @@ h3. Example
           <source nil="true"></source>
           <updated nil="true"></updated>
         </entry>
+</code></pre>
 
+h2. Copyright
 
-    p tweet        
-
-    {"truncated"=>false, "favorited"=>false, "created_at"=>"Thu Apr 15 17:01:52 +0000 2010", "text"=>"THANK GOODNESS THE LIBRARY OF CONGRESS HAS UNDERSTOOD THE IMPORTANCE OF MY TWEETS what do you mean others are getting in too", "id"=>12233609555, "in_reply_to_user_id"=>nil, "in_reply_to_screen_name"=>nil, "source"=>"<a href=\"http://apiwiki.twitter.com/\" rel=\"nofollow\">API</a>", "user"=>#<struct Titi::Provider::Twitter::User id=1468401, screen_name="sockington", protected=false, followers_count=1520814, friends_count=457, statuses_count=6234, favourites_count=2, created_at=Mon, 19 Mar 2007 03:45:00 +0000, name="Sockamillion", url="http://www.sockington.org/", location="Waltham, MA", description="I am Jason Scott's Cat.", time_zone="Eastern Time (US & Canada)", utc_offset=-18000, profile_background_color="48484c", profile_text_color="000000", profile_link_color="000000", profile_sidebar_border_color="79c021", profile_sidebar_fill_color="585e7e", profile_background_tile=false, profile_background_image_url="http://a1.twimg.com/profile_background_images/6682718/SocksTwitter.jpg", profile_image_url="http://a3.twimg.com/profile_images/77537329/IMG_3738_normal.JPG">, "in_reply_to_status_id"=>nil}
-  </code></pre>
-
-
-== Note on Patches/Pull Requests
- 
-* Fork the project.
-* Make your feature addition or bug fix.
-* Add tests for it. This is important so I don't break it in a
-  future version unintentionally.
-* Commit, do not mess with rakefile, version, or history.
-  (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
-* Send me a pull request. Bonus points for topic branches.
-
-== Copyright
-
-Copyright (c) 2010 mrflip. See LICENSE for details.
+Copyright (c) 2010 Infochimps, Inc. Available under Apache license: see LICENSE for details.

data/VERSION

@@ -1 +1 @@
-0.0.2
+0.0.3

data/lib/titi.rb

@@ -1,6 +1,4 @@
 module Titi
-  autoload :Matcher,   'titi/matcher'
-  autoload :ParseTree, 'titi/parse_tree'
   autoload :Provider,  'titi/provider'
   autoload :Adaptor,   'titi/adaptor'
 end

data/lib/titi/ignore/hpricot.rb

@@ -0,0 +1,31 @@
+require 'hpricot'
+
+module Hpricot::IMWExtensions
+  def contents_of path
+    cnts = self.at path
+    cnts.inner_html if cnts
+  end
+  def path_attr path, attr
+    cnts = self.at path
+    cnts.attributes[attr] if cnts
+  end
+  def class_of path
+    self.path_attr_safely(path, 'class')
+  end
+end
+
+class Hpricot::Elem
+  include Hpricot::IMWExtensions
+end
+
+class Hpricot::Elements
+  include Hpricot::IMWExtensions
+end
+
+class Hpricot::Doc
+  include Hpricot::IMWExtensions
+end
+
+class Hpricot::BogusETag
+  include Hpricot::IMWExtensions
+end

data/lib/titi/ignore/html_parser.rb

@@ -0,0 +1,387 @@
+#
+# h2. lib/imw/parsers/html_parser.rb -- html parser
+#
+# == About
+#
+# h4. HTML Extractor
+#
+# * map repeating HTML elements to intermediate ruby data structure
+# * optimize all the common cases for expressive brevity
+# * output structure will come from HTML structure; map to desired output objects in transform stage.
+# * spec shouldn't be allowed to get too much more complicated than this; again, transform stage exists
+#
+# If this doesn't yield satisfaction you may enjoy
+# * http://blog.labnotes.org/2006/07/11/scraping-with-style-scrapi-toolkit-for-ruby/
+# * http://scrubyt.org/
+# Note of course that these have quite different goals.  For example, we don't
+# have any interest in "interactive" crawling, eg form submission, or at least
+# that goes elsewhere.
+#
+#
+# == Sample HTML (http://twitter.com):
+#
+#   <ul class="about vcard entry-author">
+#     <li         ><span class="label">Name</span>     <span class="fn" >MarsPhoenix       </span> </li>
+#     <li         ><span class="label">Location</span> <span class="adr">Mars, Solar System</span> </li>
+#     <li id="bio"><span class="label">Bio</span>      <span class="bio">I dig Mars!       </span> </li>
+#     <li         ><span class="label">Web</span>
+#        <a href="http://tinyurl.com/5wwaru" class="url" rel="me nofollow">http://tinyurl.co...</a></li>
+#   </ul>
+#
+# == Parser Spec:
+#   :hcard        => m_one('//ul.vcard.about',
+#     {
+#       :name     => 'li/span.fn',
+#       :location => 'li/span.adr',
+#       :url      => m_attr('li/a.url[@href]', 'href'),
+#       :bio      => 'li#bio/span.bio',
+#     }
+#   )
+#
+# == Example return:
+#   { :hcard => { :name => 'Mars Phoenix', :location => 'Mars, Solar System', :bio => 'I dig Mars!', :url => 'http://tinyurl.com/5wwaru' } }
+#
+# == Sample HTML (http://delicious.com):
+#   <ul id="bookmarklist" class="bookmarks NOTHUMB">
+#     <li class="post" id="item-...">
+#       <div class="bookmark NOTHUMB">
+#         <div class="dateGroup">         <span title="23 APR 08">23 APR 08</span>     </div>
+#         <div class="data">
+#           <h4>                          <a rel="nofollow" class="taggedlink" href="http://www.cs.biu.ac.il/~koppel/BlogCorpus.htm">Blog Authorship Corpus (Blogger.com 1994)</a>
+#                                         <a class="inlinesave" href="...">SAVE</a> </h4>
+#           <h5 class="savers-label">     PEOPLE</h5>
+#           <div class="savers savers2">  <a class="delNav" href="/url/7df6661946fca61863312644eb071953"><span class="delNavCount">26</span></a>  </div>
+#           <div class="description">     The Blog Authorship Corpus consists of the collected posts of 19,320 bloggers gathered from blogger.com in August 2004. The corpus incorporates a total of 681,288 posts and over 140 million words - or approximately 35 posts and 7250 words per person. </div>
+#         </div>
+#         <div class="meta"></div>
+#         <h5 class="tag-chain-label">TAGS</h5>
+#         <div class="tagdisplay">
+#           <ul class="tag-chain">
+#             <li class="tag-chain-item off first"><a class="tag-chain-item-link" rel="tag" href="/infochimps/blog"     ><span class="tag-chain-item-span">blog</span>    </a></li>
+#             <li class="tag-chain-item off">      <a class="tag-chain-item-link" rel="tag" href="/infochimps/corpus"   ><span class="tag-chain-item-span">corpus</span>  </a></li>
+#             <li class="tag-chain-item off">      <a class="tag-chain-item-link" rel="tag" href="/infochimps/analysis" ><span class="tag-chain-item-span">analysis</span></a></li>
+#             <li class="tag-chain-item off">      <a class="tag-chain-item-link" rel="tag" href="/infochimps/nlp"      ><span class="tag-chain-item-span">nlp</span>     </a></li>
+#             <li class="tag-chain-item on  last"> <a class="tag-chain-item-link" rel="tag" href="/infochimps/dataset"  ><span class="tag-chain-item-span">dataset</span> </a></li>
+#           </ul>
+#         </div>
+#         <div class="clr"></div>
+#       </div>
+#     </li>
+#   </ul>
+#
+# == Parser Specification:
+#   :bookmarks            => [ 'ul#bookmarklist/li.post/.bookmark',
+#     {
+#       :date                     => hash(    '.dateGroup/span',
+#          [:year, :month, :day]  => regexp(  '', /(\d{2}) ([A-Z]{3}) (\d{2})/),
+#          ),
+#       :title                    =>          '.data/h4/a.taggedlink',
+#       :url                      => attr(    '.data/h4/a.taggedlink', 'href'),
+#       :del_link_url             => href(    '.data/.savers/a.delNav),
+#       :num_savers               => to_i(    '.data/.savers//span.delNavCount'),
+#       :description              =>          '.data/.description',
+#       :tags                     =>         ['.tagdisplay//tag-chain-item-span']
+#     }
+#   ]
+#
+# == Example output:
+#   { :bookmarks => [
+#     { :date             => { :year => '08', :month => 'APR', :day => '23' },
+#       :title            => 'Blog Authorship Corpus (Blogger.com 1994)',
+#       :url              => 'http://www.cs.biu.ac.il/~koppel/BlogCorpus.htm',
+#       :del_link_url     => '/url/7df6661946fca61863312644eb071953',
+#       :num_savers       => 26,
+#       :description      => 'The Blog ... ',
+#       :tags             => ['blog', 'corpus', 'analysis', 'nlp', 'dataset'],
+#      }
+#    ]}
+#
+# == Implementation:
+#
+# Internally, we take the spec and turn it into a recursive structure of Matcher
+# objects.  These consume Hpricot Elements and return the appropriately extracted
+# object.
+#
+# Note that the /default/ is for a bare selector to match ONE element, and to not
+# complain if there are many.
+#
+# Missing elements are silently ignored -- for example if
+#   :foo => 'li.missing'
+# there will simply be no :foo element in the hash (as opposed to having hsh[:foo]
+# set to nil -- hsh.include?(foo) will be false)
+#
+#
+# == List of Matchers:
+#     { :field => /spec/, ... }           # hash          hash, each field taken from spec.
+#     [ "hpricot_path" ]                  # 1-el array    array: for each element matching
+#                                                         hpricot_path, the inner_html
+#     [ "hpricot_path", /spec/ ]          # 2-el array    array: for each element matching
+#                                                         hpricot_path, pass to spec
+#     "hpricot_path"                      # string        same as one("hpricot_path")
+#     one("hpricot_path")                 # one           first match to hpricot_path
+#     one("hpricot_path", /spec/)         # one           applies spec to first match to hpricot_path
+#     (these all match on one path:)
+#     regexp("hpricot_path", /RE/)        # regexp        capture groups from matching RE against
+#                                                         inner_html of first match to hpricot_path
+#     attr("hpricot_path", 'attr_name')   # attr
+#     href("hpricot_path")                # href          shorthand for attr(foo, 'href')
+#     no_html                             #               strip tags from contents
+#     html_encoded                        #               html encode contents
+#     to_i, to_f, etc                     # convert
+#     lambda{|doc| ... }                  # proc          calls proc on current doc
+#
+# == Complicated HCard example:
+#     :hcards                     =>      [ '//ul.users/li.vcard',
+#       {
+#         :name                   =>      '.fn',
+#         :address                =>      one('.adr',
+#           :street               =>      '.street',
+#           :city                 =>      '.city',
+#           :zip                  =>      '.postal'
+#         )
+#         :tel                    =>      [ 'span.tel',
+#           {
+#             :type               =>      'span.type',
+#             [:cc, :area, :num]  =>      hp.regexp('span.value', /+(\d+).(\d{3})-(\d{3}-\d{4})/),
+#           }
+#         ]
+#         :tags                   =>      [ '.tag' ],
+#       }
+#     ]
+#
+# == Resulting Parser
+#     MatchHash({:hcards  =>      MatchArray('//ul.users/li.hcard',
+#       MatchHash({
+#         :name                   =>      MatchFirst('.fn'),
+#         :address                =>      MatchFirst('.adr',
+#           MatchHash({
+#             :street             =>      MatchFirst('.street'),
+#             :city               =>      MatchFirst('.locality),
+#             :state              =>      MatchFirst('.region),
+#             :zip                =>      MatchFirst('.postal'),
+#           }))
+#         :tel                    =>      MatchArray('span.tel',
+#           MatchHash({
+#             :type               =>      MatchFirst('span.type'),
+#             [:cc, :area, :num]  =>      RegexpMatcher('span.value', /+(\d+).(\d{3})-(\d{3}-\d{4})/),
+#           })
+#         )
+#         :tags                   =>      MatchArray('.tag'),
+#       })
+#     )
+#
+# == Example output
+#     [
+#       {:tel     => [ {:type => 'home', :cc => '49', :area => '305', :num => '555-1212'},
+#                      {:type => 'work', :cc => '49', :area => '305', :num => '555-6969'}, ],
+#        :name    => "Bob Dobbs, Jr.",
+#        :tags    => ["church"] },
+#       {:tel     => [ {:type => 'fax',  :cc => '49', :area => '305', :num => '867-5309'}, ],
+#        :name    => "Jenny",
+#        :address => { :street => "53 Evergreen Terr.", :city => "Springfield" },
+#        :tags    => ["bathroom", "wall"] },
+#     ]
+#
+# Author::    (Philip flip Kromer, Dhruv Bansal) for Infinite Monkeywrench Project (mailto:coders@infochimps.org)
+# Copyright:: Copyright (c) 2008 infochimps.org
+# License::   GPL 3.0
+# Website::   http://infinitemonkeywrench.org/
+#
+# puts "#{File.basename(__FILE__)}: Something clever" # at bottom
+
+require 'imw/parsers/html_parser/matchers'
+
+module IMW
+  module Parsers
+    class HtmlParser
+
+      include IMW::Parsers::HtmlMatchers
+
+      attr_accessor :parse_tree
+
+      #
+      # Parse Tree
+      #
+      def initialize arg_spec=nil
+        spec = arg_spec || self.class.parser_spec
+        self.parse_tree = IMW::Parsers::HtmlMatchers.build_parse_tree(spec)
+      end
+
+      #
+      # See IMW::HtmlParser for syntax
+      #
+      #
+      def self.parser_spec
+        raise "Override this to create your own parser spec"
+      end
+
+      #
+      # Walk
+      #
+      def parse doc
+        self.parse_tree.match(doc)
+      end
+
+      # one("hpricot_path")                 first match to hpricot_path
+      # one("hpricot_path", /spec/)         applies spec to first match to hpricot_path
+      #
+      def self.one selector, matcher
+        MatchFirstElement.new(selector, IMW::Parsers::HtmlMatchers.build_parse_tree(matcher))
+      end
+      # match the +attr+ attribute of the first element given by +selector+
+      def self.attr selector, attr, matcher=nil
+        MatchAttribute.new(selector, attr, IMW::Parsers::HtmlMatchers.build_parse_tree(matcher))
+      end
+      # shorthand for +attr(foo, 'href')+
+      def self.href selector, matcher=nil
+        self.attr(selector, 'href', matcher)
+      end
+      # shorthand for +attr(foo, 'src')+
+      def self.src selector, matcher=nil
+        self.attr(selector, 'src', matcher)
+      end
+
+      def self.proc selector, proc, matcher=nil
+        MatchProc.new(selector, proc, IMW::Parsers::HtmlMatchers.build_parse_tree(matcher))
+      end
+
+      # strip ","s (!! thus disrespecting locale !!!)
+      # and convert to int
+      def self.to_num selector, matcher=nil
+        proc selector, lambda{|num| num.to_s.gsub(/,/,'').to_i if num }, matcher
+      end
+      def self.to_json selector, matcher=nil
+        proc selector, lambda{|v| v.to_json if v }, matcher
+      end
+
+      def self.strip selector, matcher=nil
+        proc selector, lambda{|v| v.strip }, matcher
+      end
+
+      def self.re_group selector, re
+        MatchRegexp.new(selector, re)
+      end
+      def self.re selector, re
+        MatchRegexp.new(selector, re, nil, :capture => 1)
+      end
+      def self.re_all selector, re, matcher=nil
+        MatchRegexpRepeatedly.new(selector, re)
+      end
+
+      # def self.plain_text selector, matcher=nil
+      #   proc selector, lambda{|el| el.inner_text if el }, matcher
+      # end
+
+      # attr_accessor :mapping
+      #
+      # #
+      # # Feed me a hash and I'll semantify HTML
+      # #
+      # # The hash should magically adhere to the too-complicated,
+      # # ever evolving goatrope that works for the below
+      # #
+      # #
+      # def initialize mapping
+      #   self.mapping = mapping
+      # end
+      #
+      # #
+      # # take a document subtree,
+      # # and a mapping of hpricot paths to that subtree's data mapping
+      # # recursively extract that datamapping
+      # #
+      # def extract_tree  hdoc, content, sub_mapping
+      #   data = { }
+      #   sub_mapping.each do |selector, target|
+      #     data[selector] = []
+      #     sub_contents = content/selector
+      #     sub_contents.each do |sub_content|
+      #       sub_data = {}
+      #       extract_node hdoc, sub_content, sub_data, selector, target
+      #       data[selector] << sub_data
+      #     end
+      #   end
+      #   data
+      #   # end
+      #   #   if selector.is_a?(String)
+      #   #     conts = (content)
+      #   #   else
+      #   #     conts = [content]
+      #   #   end
+      #   #   conts[0..0].each do |content|
+      #   #     extract_node hdoc, content, data, selector, target
+      #   #   end
+      #   # end
+      #   data
+      # end
+      #
+      # #
+      # # insert the extracted element into the data mapping
+      # #
+      # def extract_node hdoc, content, data, selector, target
+      #   classification = classify_node(selector, target)
+      #   result = \
+      #   case classification
+      #   when :subtree
+      #     target.each do |sub_selector, sub_target|
+      #       extract_node hdoc, content, data, sub_selector, sub_target
+      #     end
+      #
+      #   when :sub_attribute
+      #     k, v = selector.to_a[0]
+      #     subcontent = (k[0..0] == '/') ? (hdoc.at(k)) : (content.at(k))
+      #     val  = subcontent.attributes[v.to_s] if subcontent
+      #     data[target] = val unless val.blank?
+      #
+      #   when :attribute then
+      #     val = content.attributes[selector.to_s]
+      #     data[target] = val unless val.blank?
+      #
+      #   when :flatten_list
+      #     subcontents = (selector[0..0] == '/') ? (hdoc/selector) : (content/selector)
+      #     data[target.first] = subcontents.map{|subcontent| subcontent.inner_html }
+      #
+      #   when :inner_html
+      #     subcontent = (selector[0..0] == '/') ? (hdoc.at(selector)) : (content.at(selector))
+      #     data[target] = subcontent.inner_html.strip if subcontent
+      #
+      #   else
+      #     raise "classify_node shouldn't ever return #{classification}"
+      #   end
+      #   # puts "%-19s %-19s %-31s %s" % [target.inspect[0..18], classification.inspect[0..18], selector.inspect[0..30], result.inspect[0..90]] if (classification == :sub_attribute)
+      #   # puts '' if classification == :subtree
+      # end
+      #
+      # def classify_node selector, target
+      #   case
+      #   when target.is_a?(Hash)                             then :subtree
+      #   when selector.is_a?(Hash) && (selector.length == 1) then
+      #     k, v = selector.to_a[0]
+      #     case v
+      #     when Symbol then :sub_attribute
+      #     end
+      #   when selector.is_a?(Symbol)                         then :attribute
+      #   when selector.is_a?(String) && target.is_a?(Array)  then :flatten_list
+      #   when selector.is_a?(String) && target.is_a?(Symbol) then :inner_html
+      #   else
+      #     raise "Can't classify mapping: " + [selector, target].join(" - ")
+      #   end
+      # end
+      #
+      # # use #mapping to parse file
+      # def parse link
+      #   begin       hdoc = Hpricot(link.contents)
+      #   rescue;     warn "can't hpricot #{link.to_s}" ; return false;  end
+      #   raw_taggings = extract_tree hdoc, hdoc, self.mapping
+      # end
+      #
+      # # use #mapping to parse file
+      # def parse_file filename
+      #   begin       hdoc = Hpricot(File.open(filename))
+      #   rescue;     warn "can't hpricot #{filename}" ; return false;  end
+      #   raw_taggings = extract_tree hdoc, hdoc, self.mapping
+      # end
+    end
+  end
+end
+

data/lib/titi/ignore/matcher.rb

@@ -0,0 +1,254 @@
+module Titi
+  module Matcher
+    # An abstract class from which to subclass specific HTML matchers.
+    #
+    # A subclass is initialized with a +selector+ and an optional
+    # +matcher+.  The +selector+ is an HTML path specification used to
+    # collect elements from the document.  If initialized with a
+    # +matcher+, the +matcher+ is used to return match information
+    # from the elements; else the inner HTML is returned.  Subclasses
+    # decide how the +selector+ will collect elements.
+    class Base
+
+      attr_accessor :selector
+      attr_accessor :matcher
+      attr_accessor :options
+
+      def initialize selector, matcher=nil, options={}
+        self.selector = selector
+        self.matcher  = matcher
+        self.options  = options
+      end
+
+      def match doc
+        raise "Abstract class #{self.class}"
+      end
+
+    end
+
+    # Concrete subclass of <tt>IMW::Parsers::HtmlMatchers::Matcher</tt>
+    # for matching against the first element of a document matching a
+    # selector.
+    class MatchFirstElement < Titi::Matcher::Base
+      # Grab the first element from +doc+ matching the +selector+ this
+      # class was initialized with.  If initialized with a +matcher+,
+      # then return the +matcher+'s match against the first element,
+      # else just return the inner HTML of the first element.
+      #
+      #   m = MatchFirstElement.new('span#bio/a.homepage')
+      #   m.match('<span id="bio"><a class="homepage" href="http://foo.bar">My Homepage</a></span>')
+      #   # => 'My Homepage'
+      def match doc
+        doc = Hpricot(doc) if doc.is_a?(String)
+        el = doc.at(selector) or return nil
+        if matcher
+          matcher.match(el)
+        else
+          options[:html] ? el : el.inner_text.strip
+        end
+      end
+    end
+
+    # FIXME is there really a need for this separate class?  why can't
+    # MatchFirstElement.match accept a block?
+    class MatchProc < MatchFirstElement
+      attr_accessor :proc
+      attr_accessor :options
+      def initialize selector, proc, matcher=nil, options={}
+        super selector, matcher
+        self.options = options
+        self.proc = proc
+      end
+      def match doc
+        val = super doc
+        val ? self.proc.call(val) : self.proc.call(doc)
+      end
+    end
+
+    # Concrete subclass of <tt>IMW::Parsers::HtmlMatchers::Matcher</tt>
+    # for matching each element of a document matching a selector.
+    class MatchArray < Titi::Matcher::Base
+      # Grab each element from +doc+ matching the +selector+ this
+      # class was initialized with.  If initialized with a +matcher+,
+      # then return an array consisting of the +matcher+'s match
+      # against each element, else just return an array consisting of
+      # the inner HTML of each element.
+      #
+      #   m = MatchArray.new('span#bio/a.homepage')
+      #   m.match('<span id="bio"><a class="homepage" href="http://foo.bar">My Homepage</a></span>
+      #            <span id="bio"><a class="homepage" href="http://foo.baz">Your Homepage</a></span>
+      #            <span id="bio"><a class="homepage" href="http://foo.qux">Their Homepage</a></span>')
+      #   # => ["My Homepage", "Your Homepage", "Their Homepage"]
+      def match doc
+        doc = Hpricot(doc) if doc.is_a?(String)
+        subdoc = (doc/selector) or return nil
+        if matcher
+          subdoc.map{|el| matcher.match(el)}
+        else
+          if options[:html]
+            subdoc.map{|el| el }
+          else
+            subdoc.map{|el| el.inner_text.strip }
+          end
+        end
+      end
+    end
+
+    # Concrete subclass of <tt>IMW::Parsers::HtmlMatchers::Matcher</tt>
+    # for matching an attribute of the first element of a document
+    # matching a selector.
+    class MatchAttribute < Titi::Matcher::Base
+
+      attr_accessor :attribute
+
+      # Unlike <tt>IMW::Parsers::HtmlMatchers::Matcher</tt>,
+      # <tt>IMW::Parsers::HtmlMatchers::MatchAttribute</tt> is initialized
+      # with three arguments: the +selector+ which collects elements
+      # from an HTML document, an +attribute+ to extract, and
+      # (optionally) a +matcher+ to perform the matching.
+      def initialize selector, attribute, matcher=nil
+        super selector, matcher
+        self.attribute = attribute.to_s
+      end
+
+      # Grab the first element from +doc+ matching the +selector+ this
+      # class was initialized with.  If initialized with a +matcher+,
+      # then return the +matcher+'s match against the value of the
+      # +attribute+ this class was initialized with, else just return
+      # the value of the +attribute+.
+      #
+      #   m = MatchAttribute.new('span#bio/a.homepage', 'href')
+      #   m.match('<span id="bio"><a class="homepage" href="http://foo.bar">My Homepage</a></span>')
+      #   # => 'http://foo.bar'
+      def match doc
+        doc = Hpricot(doc) if doc.is_a?(String)
+        val = doc.path_attr(selector, attribute)
+        matcher ? matcher.match(val) : val
+      end
+    end
+
+    # Concrete subclass of <tt>IMW::Parsers::HtmlMatchers::Matcher</tt>
+    # for using a regular expression to match against text in an HTML
+    # document.
+    class MatchRegexp < Titi::Matcher::Base
+
+      attr_accessor :re
+      attr_accessor :options
+
+      # Use the regular expression +re+ to return captures from the
+      # elements collected by +selector+ (treated as text) used on an
+      # HTML document (if +selector+ is +nil+ then match against the
+      # full text of the document).  If the keyword argument
+      # <tt>:capture</tt> is specified then return the corresponding
+      # group (indexing is that of regular expressions; "1" is the
+      # first capture), else return an array of all captures.  If
+      # +matcher+, then use it on the capture(s) before returning.
+      #
+      # FIXME Shouldn't the matcher come BEFORE the regexp capture,
+      # not after?
+      def initialize selector, re, matcher=nil, options={}
+        super selector, matcher
+        self.options = options
+        self.re = re
+      end
+
+      # Grab the first element from +doc+ matching the +selector+ this
+      # object was initialized with.  Use the +re+ and the (optional)
+      # capture group this object was initialized with to capture a
+      # string (or array of strings if no capture group was specified)
+      # from the collected element (treated as text). If initialized
+      # with a +matcher+, then return the +matcher+'s match against
+      # the value of the capture(s), else just return the capture(s).
+      #
+      #   m = MatchRegexp.new('span#bio/a.homepage', /Homepage of (.*)$/, nil, :capture => 1 )
+      #   m.match('<span id="bio"><a class="homepage" href="http://foo.bar">Homepage of John Chimpo</a></span>')
+      #   # => "John Chimpo"
+      def match doc
+        doc = Hpricot(doc) if doc.is_a?(String)
+        el = selector ? doc.contents_of(selector) : doc
+        m = re.match(el.to_s)
+        val = case
+              when m.nil? then nil
+              when self.options.key?(:capture) then m.captures[self.options[:capture] - 1] # -1 to match regexp indexing
+              else m.captures
+              end
+        # pass to matcher, if any
+        matcher ? matcher.match(val) : val
+      end
+    end
+
+
+    class MatchRegexpRepeatedly < Titi::Matcher::Base
+      attr_accessor :re
+      def initialize selector, re, matcher=nil
+        super selector, matcher
+        self.re = re
+      end
+      def match doc
+        doc = Hpricot(doc) if doc.is_a?(String)
+        # apply selector, if any
+        el = selector ? doc.contents_of(selector) : doc
+        return unless el
+        # get all matches
+        val = el.to_s.scan(re)
+        # if there's only one capture group, flatten the array
+        val = val.flatten if val.first && val.first.length == 1
+        # pass to matcher, if any
+        matcher ? matcher.match(val) : val
+      end
+    end
+
+    # Class for building a hash of values by using appropriate
+    # matchers against an HTML document.
+    class MatchHash
+
+      attr_accessor :match_hash
+
+      # The +match_hash+ must be a +Hash+ of symbols matched to HTML
+      # matchers (subclasses of
+      # <tt>IMW::Parsers::HtmlMatchers::Matcher</tt>).
+      def initialize match_hash
+        # Kludge? maybe.
+        raise "MatchHash requires a hash of :attributes => matchers." unless match_hash.is_a?(Hash)
+        self.match_hash = match_hash
+      end
+
+      # Use the +match_hash+ this +MatchHash+ was initialized with to
+      # select elements from +doc+ and extract information from them:
+      #
+      #   m = MatchHash.new({
+      #       :name         => MatchFirstElement.new('li/span.customer'),
+      #       :order_status => MatchAttribute.new('li/ul[@status]','status'),
+      #       :products     => MatchArray.new('li/ul/li')
+      #     })
+      #   m.match('<li><span class="customer">John Chimpo</span>
+      #                <ul status="shipped">
+      #                  <li>bananas</li>
+      #                  <li>mangos</li>
+      #                  <li>banangos</li>
+      #                </ul></li>')
+      #   # => {
+      #         :name         => "John Chimpo",
+      #         :order_status => "shipped",
+      #         :products     => ["bananas", "mangos", "banangos"]
+      #        }
+      def match doc
+        doc = Hpricot(doc) if doc.is_a?(String)
+        hsh = { }
+        match_hash.each do |attr, m|
+          val = m.match(doc)
+          case attr
+          when Array then hsh.merge!(Hash.zip(attr, val).reject{|k,v| v.nil? }) if val
+          else            hsh[attr] = val  end
+        end
+        self.class.scrub!(hsh)
+      end
+
+      # kill off keys with nil values
+      def self.scrub! hsh
+        hsh # .reject{|k,v| v.nil? }
+      end
+    end
+
+  end
+end

data/lib/titi/{parse_tree.rb → ignore/parse_tree.rb}

@@ -1,6 +1,5 @@
 module Titi
   module ParseTree
-
     #
     # construct the downstream part of a hash matcher
     #
@@ -26,7 +25,8 @@ module Titi
         MatchArray.new(spec[0].to_s, build_parse_tree(spec[1]))
       when String         then MatchFirstElement.new(spec)
       when Proc           then MatchProc.new(nil, spec)
-      when Regexp         then MatchRegexp.new(nil, spec, nil, :one => true)
+      when Regexp         then MatchRegexp.new(nil, spec, nil, :capture => 1)
+      when Symbol         then MatchAttribute.new(nil, spec, nil)
       else raise "Don't know how to parse #{spec.inspect}"
       end
     end

data/lib/titi/provider/twitter/models.rb

@@ -108,7 +108,6 @@ module Titi::Provider
       #
       def self.get status_id
         raw_json_str = RestClient.get "http://twitter.com/statuses/show/#{status_id}.json"
-        # raw_json_str = %q{{"created_at":"Mon Mar 19 21:08:24 +0000 2007","profile_sidebar_fill_color":"ffffff","description":"Founder, http://infochimps.org - Building tools to Organize, Explore and Comprehend massive data sources","verified":false,"time_zone":"Central Time (US & Canada)","status":{"in_reply_to_status_id":null,"created_at":"Sat Apr 17 05:47:28 +0000 2010","favorited":false,"in_reply_to_user_id":null,"source":"<a href=\"http://www.atebits.com/\" rel=\"nofollow\">Tweetie</a>","id":12327261220,"in_reply_to_screen_name":null,"truncated":false,"text":"Hacking Activity Streams yay"},"following":null,"profile_sidebar_border_color":"f0edd8","url":"http://infochimps.org","profile_image_url":"http://a3.twimg.com/profile_images/377919497/FlipCircle-2009-900-trans_normal.png","notifications":null,"followers_count":1061,"profile_background_color":"BCC0C8","location":"iPhone: 30.316122,-97.733817","screen_name":"mrflip","profile_background_image_url":"http://a3.twimg.com/profile_background_images/2348065/2005Mar-AustinTypeTour-075_-_Rappers_Delight_Raindrop.jpg","friends_count":754,"statuses_count":1744,"profile_text_color":"000000","protected":false,"profile_background_tile":false,"favourites_count":98,"name":"Philip Flip Kromer","contributors_enabled":false,"profile_link_color":"0000ff","id":1554031,"lang":"en","geo_enabled":true,"utc_offset":-21600}}
         raw_status = JSON.load(raw_json_str.to_s)
         adapt(raw_status)
       end

data/titi.gemspec

@@ -5,7 +5,7 @@
 
 Gem::Specification.new do |s|
   s.name = %q{titi}
-  s.version = "0.0.2"
+  s.version = "0.0.3"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["mrflip"]
@@ -13,21 +13,23 @@ Gem::Specification.new do |s|
   s.description = %q{Facade adapting apis to activity streams spec}
   s.email = %q{flip@infochimps.org}
   s.extra_rdoc_files = [
-    "LICENSE",
+    "LICENSE.textile",
      "README.textile"
   ]
   s.files = [
     ".document",
      ".gitignore",
-     "LICENSE",
+     "LICENSE.textile",
      "README.textile",
      "Rakefile",
      "VERSION",
      "examples/twitter.rb",
      "lib/titi.rb",
      "lib/titi/adaptor.rb",
-     "lib/titi/matcher.rb",
-     "lib/titi/parse_tree.rb",
+     "lib/titi/ignore/hpricot.rb",
+     "lib/titi/ignore/html_parser.rb",
+     "lib/titi/ignore/matcher.rb",
+     "lib/titi/ignore/parse_tree.rb",
      "lib/titi/provider.rb",
      "lib/titi/provider/README.textile",
      "lib/titi/provider/activity_streams.rb",

metadata

@@ -5,8 +5,8 @@ version: !ruby/object:Gem::Version
   segments: 
   - 0
   - 0
-  - 2
-  version: 0.0.2
+  - 3
+  version: 0.0.3
 platform: ruby
 authors: 
 - mrflip
@@ -92,20 +92,22 @@ executables: []
 extensions: []
 
 extra_rdoc_files: 
-- LICENSE
+- LICENSE.textile
 - README.textile
 files: 
 - .document
 - .gitignore
-- LICENSE
+- LICENSE.textile
 - README.textile
 - Rakefile
 - VERSION
 - examples/twitter.rb
 - lib/titi.rb
 - lib/titi/adaptor.rb
-- lib/titi/matcher.rb
-- lib/titi/parse_tree.rb
+- lib/titi/ignore/hpricot.rb
+- lib/titi/ignore/html_parser.rb
+- lib/titi/ignore/matcher.rb
+- lib/titi/ignore/parse_tree.rb
 - lib/titi/provider.rb
 - lib/titi/provider/README.textile
 - lib/titi/provider/activity_streams.rb

data/LICENSE

@@ -1,20 +0,0 @@
-Copyright (c) 2009 mrflip
-
-Permission is hereby granted, free of charge, to any person obtaining
-a copy of this software and associated documentation files (the
-"Software"), to deal in the Software without restriction, including
-without limitation the rights to use, copy, modify, merge, publish,
-distribute, sublicense, and/or sell copies of the Software, and to
-permit persons to whom the Software is furnished to do so, subject to
-the following conditions:
-
-The above copyright notice and this permission notice shall be
-included in all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
-LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
-OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
-WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/lib/titi/matcher.rb

@@ -1,157 +0,0 @@
-module Titi
-  module Matcher
-    class Base
-      attr_accessor :selector
-      attr_accessor :matcher
-      def initialize selector, matcher=nil
-        self.selector = selector
-        self.matcher  = matcher
-      end
-      def match doc
-        raise "Abstract class #{self.class}"
-      end
-    end
-
-    class MatchFirstElement < Base
-      # return the inner_html of the given element
-      #
-      # eg.
-      #   m = MatchFirstElement.new('span#bio/a.homepage')
-      #   m.match('<span id="bio"><a class="homepage" href="http://foo.bar">My Homepage</a></span>')
-      #   # => 'My Homepage'
-      #
-      def match doc
-        el = doc.at(selector) or return nil
-        matcher ? matcher.match(el) : el.inner_html
-      end
-    end
-
-    class MatchArray < Base
-      # return the given element
-      #
-      # eg.
-      #   m = MatchAttribute.new('span#bio/a.homepage', 'href')
-      #   m.match('<span id="bio"><a class="homepage" href="http://foo.bar">My Homepage</a></span>')
-      #   # => 'http://foo.bar'
-      #
-      def match doc
-        subdoc = (doc/selector) or return nil
-        if matcher
-          subdoc.map{|el| matcher.match(el)}
-        else
-          subdoc.map{|el| el.inner_html}
-        end
-      end
-    end
-
-    class MatchAttribute < Base
-      attr_accessor :attribute
-      def initialize selector, attribute, matcher=nil
-        super selector, matcher
-        self.attribute = attribute.to_s
-      end
-      # return the given attribute on the first matching path
-      #
-      # eg.
-      #   m = MatchAttribute.new('span#bio/a.homepage', 'href')
-      #   m.match('<span id="bio"><a class="homepage" href="http://foo.bar">My Homepage</a></span>')
-      #   # => 'http://foo.bar'
-      #
-      def match doc
-        val = doc.path_attr(selector, attribute)
-        matcher ? matcher.match(val) : val
-      end
-    end
-
-    #
-    # map html elements -- or any HTMLParser tree -- to attributes in a hash.
-    #   HTMLParser.new([ {
-    #       :name     => 'li/span.fn',
-    #       :location => 'li/span.adr',
-    #       :url      => HTMLParser.attr('li/a.url[@href]', 'href'),
-    #       :bio      => 'li#bio/span.bio',
-    #     }
-    #   ])
-    #
-    class MatchHash
-      attr_accessor :match_hash
-      def initialize match_hash
-        # Kludge? maybe.
-        raise "MatchHash requires a hash of :attributes => matchers." unless match_hash.is_a?(Hash)
-        self.match_hash = match_hash
-      end
-      # Returns a hash mapping each attribute in match_hash
-      # to the result of its matcher on the current doc tree
-      def match doc
-        hsh = { }
-        match_hash.each do |attr, m|
-          val = m.match(doc)
-          case attr
-          when Array then hsh.merge!(Hash.zip(attr, val).reject{|k,v| v.nil? }) if val
-          else            hsh[attr] = val  end
-        end
-        self.class.scrub!(hsh)
-      end
-      # kill off keys with nil values
-      def self.scrub! hsh
-        hsh # .reject{|k,v| v.nil? }
-      end
-    end
-
-    class MatchProc < MatchFirstElement
-      attr_accessor :proc
-      attr_accessor :options
-      def initialize selector, proc, matcher=nil, options={}
-        super selector, matcher
-        self.options = options
-        self.proc = proc
-      end
-      def match doc
-        val = super doc
-        self.proc.call(val) if val
-      end
-    end
-
-    class MatchRegexp < Base
-      attr_accessor :re
-      attr_accessor :options
-      def initialize selector, re, matcher=nil, options={}
-        super selector, matcher
-        self.options = options
-        self.re = re
-      end
-      def match doc
-        # apply selector, if any
-        el = selector ? doc.contents_of(selector) : doc
-        m = re.match(el.to_s)
-        val = case
-        when m.nil? then nil
-        when self.options[:one] then m.captures.first
-        else m.captures
-        end
-        # pass to matcher, if any
-        matcher ? matcher.match(val) : val
-      end
-    end
-
-    class MatchRegexpRepeatedly < Base
-      attr_accessor :re
-      def initialize selector, re, matcher=nil
-        super selector, matcher
-        self.re = re
-      end
-      def match doc
-        # apply selector, if any
-        el = selector ? doc.contents_of(selector) : doc
-        return unless el
-        # get all matches
-        val = el.to_s.scan(re)
-        # if there's only one capture group, flatten the array
-        val = val.flatten if val.first && val.first.length == 1
-        # pass to matcher, if any
-        matcher ? matcher.match(val) : val
-      end
-    end
-
-  end
-end