libcraigscrape 0.5

data/COPYING.LESSER

@@ -0,0 +1,165 @@
+		   GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+  This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+  0. Additional Definitions.
+
+  As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+  "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+  An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+  A "Combined Work" is a work produced by combining or linking an
+Application with the Library.  The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+  The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+  The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+  1. Exception to Section 3 of the GNU GPL.
+
+  You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+  2. Conveying Modified Versions.
+
+  If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+   a) under this License, provided that you make a good faith effort to
+   ensure that, in the event an Application does not supply the
+   function or data, the facility still operates, and performs
+   whatever part of its purpose remains meaningful, or
+
+   b) under the GNU GPL, with none of the additional permissions of
+   this License applicable to that copy.
+
+  3. Object Code Incorporating Material from Library Header Files.
+
+  The object code form of an Application may incorporate material from
+a header file that is part of the Library.  You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+   a) Give prominent notice with each copy of the object code that the
+   Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the object code with a copy of the GNU GPL and this license
+   document.
+
+  4. Combined Works.
+
+  You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+   a) Give prominent notice with each copy of the Combined Work that
+   the Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the Combined Work with a copy of the GNU GPL and this license
+   document.
+
+   c) For a Combined Work that displays copyright notices during
+   execution, include the copyright notice for the Library among
+   these notices, as well as a reference directing the user to the
+   copies of the GNU GPL and this license document.
+
+   d) Do one of the following:
+
+       0) Convey the Minimal Corresponding Source under the terms of this
+       License, and the Corresponding Application Code in a form
+       suitable for, and under terms that permit, the user to
+       recombine or relink the Application with a modified version of
+       the Linked Version to produce a modified Combined Work, in the
+       manner specified by section 6 of the GNU GPL for conveying
+       Corresponding Source.
+
+       1) Use a suitable shared library mechanism for linking with the
+       Library.  A suitable mechanism is one that (a) uses at run time
+       a copy of the Library already present on the user's computer
+       system, and (b) will operate properly with a modified version
+       of the Library that is interface-compatible with the Linked
+       Version.
+
+   e) Provide Installation Information, but only if you would otherwise
+   be required to provide such information under section 6 of the
+   GNU GPL, and only to the extent that such information is
+   necessary to install and execute a modified version of the
+   Combined Work produced by recombining or relinking the
+   Application with a modified version of the Linked Version. (If
+   you use option 4d0, the Installation Information must accompany
+   the Minimal Corresponding Source and Corresponding Application
+   Code. If you use option 4d1, you must provide the Installation
+   Information in the manner specified by section 6 of the GNU GPL
+   for conveying Corresponding Source.)
+
+  5. Combined Libraries.
+
+  You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+   a) Accompany the combined library with a copy of the same work based
+   on the Library, uncombined with any other library facilities,
+   conveyed under the terms of this License.
+
+   b) Give prominent notice with the combined library that part of it
+   is a work based on the Library, and explaining where to find the
+   accompanying uncombined form of the same work.
+
+  6. Revised Versions of the GNU Lesser General Public License.
+
+  The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+  Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+  If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.

data/README

@@ -0,0 +1,69 @@
+= libcraigscrape - A craigslist URL-scraping support Library
+
+An easy library to do the heavy lifting between you and Craigslist's posting database. Given a URL, libcraigscrape will
+follow links, scrape fields, and make ruby-sense out of the raw html from craigslist's servers.
+
+For more information, head to the {craiglist monitoring}[http://www.derosetechnologies.com/community/libcraigscrape] help section of our website.
+
+== craigwatch
+libcraigscrape was primarily developed to support the included craigwatch[link:files/bin/craigwatch.html] script. See the included craigwatch script for
+examples of libcraigscape in action, and (hopefully) to serve an immediate craigscraping need.
+
+== Installation
+
+Install via RubyGems:
+
+  sudo gem install libcraigscrape
+
+== Usage
+
+=== Scrape Craigslist Listings since Apr 26
+
+Using the search url http://miami.craigslist.org/search/sss?query=apple
+
+  require 'libcraigscrape'
+  require 'date'
+  require 'pp'
+  
+  posts = CraigScrape.scrape_posts_since 'http://miami.craigslist.org/search/sss?query=apple', Time.parse('Apr 25')
+  posts.each do |post|
+    pp post
+  end
+
+=== Scrape Last 225 Craigslist Listings
+
+Under the category url http://miami.craigslist.org/apa/
+
+  require 'libcraigscrape'
+  require 'pp'
+  
+  posts = CraigScrape.scrape_posts 'http://miami.craigslist.org/apa/', 225
+  posts.each do |post|
+    pp post
+  end
+
+=== Scrape Single Craigslist Posting
+
+This grabs the full details under the specific post http://miami.craigslist.org/mdc/sys/1140808860.html
+
+  require 'libcraigscrape'
+  
+  post = CraigScrape.scrape_full_post 'http://miami.craigslist.org/mdc/sys/1140808860.html'
+  puts "(%s) %s:\n %s" % [ post.post_time.strftime('%b %d'), post.title, post.contents_as_plain ]
+
+=== Scrape Single Craigslist Listing
+
+This grabs the post summaries of the single listings at http://miami.craigslist.org/search/sss?query=laptop
+
+  require 'libcraigscrape'
+  
+  listing = CraigScrape.scrape_listing 'http://miami.craigslist.org/search/sss?query=laptop'
+  puts 'Found %d posts for the search "laptop" on this page' % listing.posts.length
+
+== Author
+- Chris DeRose (cderose@derosetechnologies.com)
+- DeRose Technologies, Inc. http://www.derosetechnologies.com
+
+== License
+
+See COPYING[link:files/COPYING.html]

data/Rakefile

@@ -0,0 +1,72 @@
+require 'rake'
+require 'rake/clean'
+require 'rake/gempackagetask'
+require 'rake/rdoctask'
+require 'rake/testtask'
+require 'fileutils'
+include FileUtils
+
+RbConfig = Config unless defined? RbConfig
+
+NAME = "libcraigscrape"
+VERS = ENV['VERSION'] || "0.5"
+PKG = "#{NAME}-#{VERS}"
+
+RDOC_OPTS = ['--quiet', '--title', 'The libcraigscrape Reference', '--main', 'README', '--inline-source']
+PKG_FILES = %w(CHANGELOG COPYING COPYING.LESSER Rakefile) + Dir.glob("{bin,test,lib}/**/*")
+
+SPEC =
+  Gem::Specification.new do |s|
+    s.name = NAME
+    s.version = VERS
+    s.platform = Gem::Platform::RUBY
+    s.has_rdoc = true
+    s.bindir = 'bin'
+    s.executables = 'craigwatch'
+    s.rdoc_options += RDOC_OPTS
+    s.extra_rdoc_files = ['README', "CHANGELOG", "COPYING","COPYING.LESSER", 'bin/craigwatch']
+    s.summary = "quick, easy, craigslist parsing library that takes the monotony out of working with craigslist posts and listings"
+    s.description = s.summary
+    s.author = "Chris DeRose, DeRose Technologies, Inc."
+    s.email = 'cderose@derosetechnologies.com'
+    s.homepage = 'http://www.derosetechnologies.com/community/libcraigscrape'
+    s.rubyforge_project = 'libcraigwatch'
+    s.files = PKG_FILES
+    s.require_paths = ["lib"] 
+    s.test_files = FileList['test/test_*.rb']
+    s.add_dependency 'hpricot'
+    s.add_dependency 'htmlentities'
+    s.add_dependency 'activesupport'
+  end
+
+desc "Run all the tests"
+Rake::TestTask.new do |t|
+    t.libs << "test"
+    t.test_files = FileList['test/test_*.rb']
+    t.verbose = true
+end
+
+Rake::RDocTask.new do |rdoc|
+    rdoc.rdoc_dir = 'doc/rdoc'
+    rdoc.options += RDOC_OPTS
+    rdoc.main = "README"
+    rdoc.rdoc_files.add ['README', 'CHANGELOG', 'COPYING', 'COPYING.LESSER', 'bin/craigwatch','lib/**/*.rb']
+end
+
+Rake::GemPackageTask.new(SPEC) do |p|
+    p.need_tar = true
+    p.gem_spec = SPEC
+end
+
+task "lib" do
+  directory "lib"
+end
+
+task :install do
+  sh %{rake package}
+  sh %{sudo gem install pkg/#{NAME}-#{VERS}}
+end
+
+task :uninstall => [:clean] do
+  sh %{sudo gem uninstall #{NAME}}
+end

data/bin/craig_report_schema.yml

@@ -0,0 +1,57 @@
+type: map
+class: CraigReportDefinition
+mapping:
+   "debug_database":    { type: bool, required: no }
+   "debug_mailer":      { type: bool, required: no }
+   "debug_craigscrape": { type: bool, required: no }
+
+   "report_name":    { type: str, required: yes }
+   "email_to":       { type: str, required: yes }
+   "email_from":     { type: str, required: no }
+   "smtp_settings":
+      type: map
+      required: no
+      mapping:
+         "address":         { type: str, required: yes }
+         "port":            { type: int, required: no, default: 25 }
+         "user_name":       { type: str, required: no }
+         "domain":          { type: str, required: no }
+         "password":        { type: str, required: no }
+         "authentication":  { type: str, required: no }
+   "tracking_database":
+      type: map
+      mapping:
+         "adapter":  { type: str, required: yes }
+         "dbfile":   { type: str, required: no }
+         "host":  	{ type: str, required: no }
+         "username": { type: str, required: no }
+         "password": { type: str, required: no }
+         "socket":   { type: str, required: no }
+         "database": { type: str, required: no }
+   "searches":
+      type:  seq
+      sequence:
+         - type: map
+           class: CraigReportDefinition::SearchDefinition
+           mapping:
+              "name":                {type: str,  required: yes, unique: yes}
+              "has_image":           {type: bool, required: no}
+              "newest_first":        {type: bool, required: no, default: no}
+              "price_required":      {type: bool, required: no, default: no}
+              "price_greater_than":  {type: int, required: no}
+              "price_less_than":     {type: int, required: no}
+              "full_post_has":       {type: seq, required: no, sequence: [ {type: str, unique: yes} ]}
+              "full_post_has_no":    {type: seq, required: no, sequence: [ {type: str, unique: yes} ]}
+              "summary_post_has":    {type: seq, required: no, sequence: [ {type: str, unique: yes} ]}
+              "summary_post_has_no": {type: seq, required: no, sequence: [ {type: str, unique: yes} ]}
+              "listing":   
+                 type: seq
+                 required: yes
+                 sequence:
+                   - type: str
+                     pattern: /^http[s]?\:\/\//
+                     unique: yes
+              "starting":  
+                 type: str
+                 required: no
+                 pattern: /^[\d]{1,2}\/[\d]{1,2}\/(?:[\d]{2}|[\d]{4})$/

data/bin/craigwatch

@@ -0,0 +1,374 @@
+#!/usr/bin/ruby
+#
+# =craigwatch - A email-based "post monitoring" solution
+#
+# Created alongside the libcraigscrape library, libcraigwatch was designed to take the monotony out of regular 
+# craiglist monitoring. craigwatch is designed to be run at periodic intervals (hourly/daily/etc) through crontab 
+# and report all new postings within a listing or search url, since its last run, by email.
+#
+# For more information, head to the {craiglist monitoring}[http://www.derosetechnologies.com/community/libcraigscrape] help section of our website.
+#
+# == Features
+# In additon to its report tracking, craigwatch offers many post search and filtering options that offer much imrpoved
+# and more accurate results then does craigslist's search functions. Post filtering options include:
+# - has_image - yes/no
+# - price_required - yes/no
+# - price_greater_than - (int)
+# - price_less_than - (int)
+# - full_post_has - (string or regexp) Only post whose full-post's contents contains/matches
+# - full_post_has_no - (string or regexp) Only post whose full-post's contents_ contains doesn't contain/match
+# - summary_post_has - (string or regexp) Only post whose listing's label contains/matches
+# - summary_post_has_no - (string or regexp) Only post whose listing's label doesn't contain/match
+#
+# Multiple searches can be combined into a single report, and results can be sorted by newest-first or oldest-first (default)
+# 
+# Reporting output is easily customized html, handled by ActionMailer, and emails can be delivered via smtp or sendmail.
+# Database tracking of already-delivered posts is handled by ActiveRecord, and its driver-agnostic SQL supports all the 
+# major backends (sqllite/mysql/postgres/probably-all-others). Database sizes are contained by automatically pruning old results
+# that are no longer required at the end of each run.
+#
+# Pretty useful, no?
+# 
+# == Installation
+# craigwatch is coupled with libcraigscrape, and is installed via ruby gems. However, since we focused on keeping the 
+# libcraigscrape download 'lightweight' some additional gems need to be installed in addition to the initial libcraigscrape
+# gem itself. 
+#
+# This should take care of the craigwatch install on all systems:
+#    sudo gem install libcraigscrape kwalify activerecord actionmailer
+# Alternatively, if you've already installed libcraigscrape and want to start working with craigwatch:
+#    sudo gem install kwalify activerecord actionmailer
+# 
+# This script was initially developed with activerecord 2.3, actionmailer 2.3 and kwalify 0.7, but will likely work with most 
+# prior and future versions of these libraries.
+# 
+# == Usage
+# When craigwatch is invoked, it is designed to run a single report and then terminate. There is only one parameter to craigwatch, and 
+# this parameter is the path to a valid report-definition yml file. ie:
+#    craigwatch johns_daily_watch.yml
+#
+# There is an included kwalify schema which can validate your definition files, but craigwatch will automatically do so at startup.
+# Probably, the best way to understand the report definition files, is to look at the annotated sample file below, and use it as a
+# starting point for your own.
+#
+# By default there is no program output, however, setting any of the following paramters to 'yes' in your definition file will turn on
+# useful debugging/logging output:
+# - debug_database
+# - debug_mailer
+# - debug_craigscrape
+#
+# == Definition File Sample
+# 
+# Here's a simple annotated report which uses most of the available craigwatch features:
+#
+#    # The report_name is fed into Time.now.strftime, hence the formatting characters
+#    report_name: Craig Watch For Johnathan on %D at %I:%M %p
+#
+#    email_to: Johnathan Peabody <john@example.local>
+#
+#    # This is sent straight into ActiveRecord, so there's plenty of options available here. the following is an easy
+#    # default sqlite store that should work on most any system with a minimal overhead
+#    tracking_database: { adapter: sqlite3, dbfile: /home/john/john_cwatch_report.db }
+#
+#    searches:
+#       # Search #1:
+#       - name: Schwinn Bikes For Sale in/near New York
+#
+#         # Scrape the following listings pages:
+#         listing:
+#            - http://newyork.craigslist.org/bik/
+#            - http://newyork.craigslist.org/jsy/bik/
+#         # This starting date is mostly for the first run, and gives us a reasonable cut-off point from whcih to build
+#         starting: 5/2/2009
+#
+#         # We want listings with Schwinn in the summary
+#         summary_post_has: [ /schwinn/i ]
+#
+#         # We're only interested in adult bikes, so scrap any results that mentions chidren or kids
+#         full_post_has_no: [ /(children|kids)/i ]
+#
+#         # Oh, and we're on a budget:
+#         price_less_than: 120
+#     
+#       # Search #2
+#       - name: Large apartment rentals in San Francisco
+#         # We're going to rely on craigslist's built-in search for this one since there's a lot of listings, and we 
+#         # want to conserve some bandwidth
+#         listing: [ http://sfbay.craigslist.org/search/apa?query=pool&minAsk=min&maxAsk=max&bedrooms=5 ]
+#         starting: 5/2/2009
+#         # We'll require a price to be listed, 'cause it keeps out some of the unwanted fluff
+#         price_required: yes
+# 
+#         # Hopefully this will keep us away from a bad part of town:
+#         price_greater_than: 1000        
+#
+#         # Since we dont have time to driv to each location, we'll require only listings with pictures
+#         has_image: yes
+#
+# == Author
+# - Chris DeRose (cderose@derosetechnologies.com)
+# - DeRose Technologies, Inc. http://www.derosetechnologies.com
+#
+# == License
+#
+# See COPYING[link:files/COPYING.html]
+#
+$: << File.dirname(__FILE__) + '/../lib'
+
+require 'rubygems'
+require 'kwalify'
+require 'kwalify/util/hashlike'
+require 'active_record'
+require 'action_mailer'
+require 'libcraigscrape'
+require "socket"
+
+class String #:nodoc:
+  RE = /^\/(.*)\/([ixm]*)$/
+  
+  def is_re?
+    (RE.match self) ? true : false
+  end
+  
+  def to_re
+    source, options = ( RE.match(self) )? [$1, $2] : [self,nil]
+    mods = 0
+
+    options.each_char do |c| 
+      mods |= case c
+        when 'i': Regexp::IGNORECASE
+        when 'x': Regexp::EXTENDED
+        when 'm': Regexp::MULTILINE
+      end
+    end unless options.nil? or options.empty?
+
+    Regexp.new source, mods
+  end
+end
+
+class CraigReportDefinition #:nodoc:
+  include Kwalify::Util::HashLike
+
+  attr_reader :report_name, :email_to, :email_from, :tracking_database, :searches, :smtp_settings
+
+  def debug_database?;    @debug_database; end
+  def debug_mailer?;      @debug_mailer; end
+  def debug_craigscrape?; @debug_craigscrape; end
+    
+  def each_search(&block); searches.each &block; end
+
+  def email_from
+    (@email_from) ? @email_from : ('%s@%s' % [ENV['USER'], Socket.gethostname])
+  end
+
+  class SearchDefinition #:nodoc:
+    include Kwalify::Util::HashLike   
+    
+    attr_reader :name, :listing
+    attr_reader :full_post_has, :full_post_has_no
+    attr_reader :summary_post_has, :summary_post_has_no
+    
+    attr_reader :price_greater_than,:price_less_than
+
+    def has_image?; @has_image; end
+    def newest_first?; @newest_first; end
+    def price_required?; @price_required; end
+      
+    def starting_at
+      (@starting) ? 
+        Time.parse(@starting) : 
+        Time.now.yesterday.beginning_of_day
+    end
+    
+    def passes_filter?(post)  
+      if post.price.nil?
+        return false if price_required?
+      else
+        return false if @price_greater_than and post.price <= @price_greater_than
+        return false if @price_less_than and post.price >= @price_less_than
+      end
+      
+      return false unless matches_all? summary_post_has, post.label
+      return false unless doesnt_match_any? summary_post_has_no, post.label
+      
+      if full_post_has or full_post_has_no
+        # We're going to download the page, so let's make sure we didnt hit a "This posting has been flagged for removal"
+        return false if post.full_post.title.nil?
+        
+        return false unless matches_all? full_post_has, post.full_post.contents_as_plain
+        return false unless doesnt_match_any? full_post_has_no, post.full_post.contents_as_plain
+      end
+
+      true
+    end
+    
+    private
+    
+    def matches_all?(conditions, against)
+      (conditions.nil? or conditions.all?{|c| match_against c, against}) ? true : false
+    end
+    
+    def doesnt_match_any?(conditions, against)
+      (conditions.nil? or conditions.all?{|c| !match_against c, against}) ? true : false
+    end
+    
+    def match_against(condition, against)
+      (against.scan( condition.is_re? ? condition.to_re : condition).length > 0) ? true : false
+    end
+  end
+end
+
+class TrackedSearch < ActiveRecord::Base #:nodoc:
+  has_many :tracked_posts, :dependent => :destroy
+  validates_uniqueness_of :search_name
+  validates_presence_of   :search_name
+  
+  def already_tracked?(url)
+    ( self.tracked_posts.find :first, :conditions => ['url = ?', url]) ? true : false
+  end
+  
+  def last_tracked_at
+    self.tracked_posts.maximum 'created_at'
+  end
+end
+
+class TrackedPost < ActiveRecord::Base #:nodoc:
+  belongs_to :tracked_search
+  validates_presence_of :url, :tracked_search_id
+  validates_uniqueness_of :url, :scope => :tracked_search_id
+end
+
+class ReportMailer < ActionMailer::Base #:nodoc:
+  def report(to, sender, subject_template, report_tmpl)
+    
+    formatted_subject = Time.now.strftime(subject_template)
+    
+    recipients  to
+    from        sender
+    subject     formatted_subject
+
+    generate_view_parts 'craigslist_report', report_tmpl.merge({:subject =>formatted_subject})
+  end
+
+  def generate_view_parts(view_name, tmpl)
+    part( :content_type => "multipart/alternative" ) do |p|
+      [
+        { :content_type => "text/plain", :body => render_message("#{view_name.to_s}.plain.erb", tmpl) },
+        { :content_type => "text/html",  :body => render_message("#{view_name.to_s}.html.erb",  tmpl.merge({:part_container => p})) }
+      ].each { |parms| p.part parms.merge( { :charset => "UTF-8", :transfer_encoding => "7bit" } ) }
+    end
+  end
+end
+
+#############
+
+# Let's start our program now:
+report_definition_file = ARGV[0] if ARGV[0] and File.readable?(ARGV[0])
+
+unless report_definition_file
+  puts <<EOD
+Usage:
+    #{File.basename($0)} [report_definition_file]
+    
+Run 'gem server' and browse the libcraigscrape rdoc for 'bin/craigscrape' for specific usage details.
+EOD
+  exit
+end
+
+# Validate/Parse our input file:
+parser = Kwalify::Yaml::Parser.new(
+  Kwalify::Validator.new(
+    Kwalify::Yaml.load_file(File.dirname(__FILE__)+'/craig_report_schema.yml')
+  )
+)
+parser.data_binding = true
+
+craig_report = parser.parse_file report_definition_file
+
+parser.errors.each do |e|
+  puts "Definition Validation Error (line #{e.linenum}, char #{e.column}): #{e.message}"
+end and exit if parser.errors.length > 0
+
+# Initialize Action Mailer:
+ActionMailer::Base.logger = Logger.new STDERR if craig_report.debug_mailer?
+if craig_report.smtp_settings
+  ReportMailer.smtp_settings = craig_report.smtp_settings
+else
+  ReportMailer.delivery_method = :sendmail  
+end
+ReportMailer.template_root = File.dirname __FILE__
+
+# Initialize the database:
+ActiveRecord::Base.logger = Logger.new STDERR if craig_report.debug_database?
+ActiveRecord::Base.establish_connection craig_report.tracking_database
+
+# Initialize CraigScrape (sorta)
+CraigScrape.logger = Logger.new STDERR if craig_report.debug_craigscrape?
+
+# Perform migrations if needed?
+ActiveRecord::Schema.define do
+  suppress_messages do 
+    create_table :tracked_searches do |t|
+      t.column :search_name,      :string
+    end unless table_exists? :tracked_searches
+  
+    create_table :tracked_posts do |t|
+      t.column :url,                :string
+      t.column :tracked_search_id,  :integer
+      t.column :created_at,         :date
+    end unless table_exists? :tracked_posts
+  end
+end
+
+# Now let's run a report:
+report_summaries = []
+
+craig_report.each_search do |search|
+
+  # Load our tracking info
+  search_track = TrackedSearch.find :first, :conditions => ['search_name = ?',search.name]
+  
+  # No Tracking found - let's set one up:
+  search_track = TrackedSearch.create! :search_name => search.name unless search_track
+
+  last_tracked_at = (search_track.last_tracked_at) ? search_track.last_tracked_at : search.starting_at
+
+  # Let's collect all the summaries that could apply:
+  new_summaries = {}
+  search.listing.each do |listing|
+    CraigScrape.scrape_posts_since(listing, last_tracked_at).each do |p_s|
+      new_summaries[p_s.full_url] = p_s unless new_summaries.has_key? p_s.full_url or search_track.already_tracked? p_s.full_url
+    end
+  end
+  
+  # Let's flatten the unique'd hash into a more useable array:
+  new_summaries = new_summaries.values.sort{|a,b| a.date <=> b.date} # oldest goes to bottom
+  
+  # Reject anything which doesn't match the has/has_no :
+  new_summaries.reject!{|s| !search.passes_filter? s }
+
+  # Now Let's manage the tracking database:
+  if new_summaries.length > 0  
+    # Insert all the relevant, newest tracked posts: 
+    # NOTE: Since all the dates are at_begining_of_day, we'll effectively have a chunk of dates tied for latest
+    new_summaries.reject{|p| p.date < new_summaries.last.date}.each do |p_s|
+      search_track.tracked_posts.create! :url => p_s.full_url, :created_at => p_s.date
+    end
+  
+    # Remove all tracked posts older then we could need:
+    TrackedPost.delete_all ['tracked_search_id = ? AND created_at < ?', search_track.id, new_summaries.last.date ]
+    
+    new_summaries.reverse! if search.newest_first?
+    
+    # We'll want to email these...
+    report_summaries << { :postings => new_summaries, :search => search }
+  end
+end
+
+# Time to send the email:
+ReportMailer.deliver_report(
+  craig_report.email_to, 
+  craig_report.email_from, 
+  craig_report.report_name, 
+  {:summaries => report_summaries, :definition => craig_report}
+) if report_summaries.length > 0