flatfish 0.3.1

data/.gitignore

@@ -0,0 +1,2 @@
+flatfish*gem
+*.swp

data/Gemfile

@@ -0,0 +1,7 @@
+source 'http://rubygems.org'
+
+gem 'nokogiri'
+gem 'activerecord'
+gem 'mysql2'
+
+gem 'awesome_print'

data/Gemfile.lock

@@ -0,0 +1,31 @@
+GEM
+  remote: http://rubygems.org/
+  specs:
+    activemodel (3.2.6)
+      activesupport (= 3.2.6)
+      builder (~> 3.0.0)
+    activerecord (3.2.6)
+      activemodel (= 3.2.6)
+      activesupport (= 3.2.6)
+      arel (~> 3.0.2)
+      tzinfo (~> 0.3.29)
+    activesupport (3.2.6)
+      i18n (~> 0.6)
+      multi_json (~> 1.0)
+    arel (3.0.2)
+    awesome_print (1.0.2)
+    builder (3.0.0)
+    i18n (0.6.0)
+    multi_json (1.3.6)
+    mysql2 (0.3.11)
+    nokogiri (1.5.5)
+    tzinfo (0.3.33)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  activerecord
+  awesome_print
+  mysql2
+  nokogiri

data/HACKING.md

@@ -0,0 +1,25 @@
+# HACKING FLATFISH 
+
+## Hello Brave Soul
+Thank you for your interest in contributing to Flatfish!
+
+### Background
+Flatfish is built on Rails (Active Support + Active Record) and Nokogiri.  The first thing to note is that many Rails helpers (but not all) are available, but Flatfish is not a webapp--the directory structure is different, there is no MVC, etc.  So you will find that Flatfish is *Rails-like*, but does some hackery like creating dynamic models.  Active Record enables Flatfish to parse a CSV and save a data model representing it on-the-fly.
+
+Nokogiri is an XML/HTML parser that supports CSS Selectors--basically, you get jQuery in Ruby.  It's awesome. Flatfish primarily uses Nokogiri for 1) specificity, as in only grabbing the HTML we want and 2) updating the links, images, and files.  Links, images, and files have their paths corrected and tokenized: additionally, files and images are saved as blobs in the database.
+
+### Basic Process Flow
+A Flatfish object is instantiated with a YAML config file.  The YAML includes DB info, some general options, and specifics on the Types of web pages to be processed.  Each Type has a CSV and URL Host.  A table for each Type is created if necessary and an Active Record subclass is generated dynamically.  Each row of the CSV is parsed, allowing Flatfish to:
+1. Grab the HTML from a remote host or a local directory
+2. Attempt to handle both HTTPS Redirects and Basic Authentication
+3. Correct all link, image, and file paths
+4. Tokenize all links, images, and files
+5. Update or create all files and images saving them to the DB--eg, multiple copies of the same image are possible with unique URLs
+6. Update or create the HTML saving it to the DB--again, the URL is the unique identifier
+
+### Odds and Ends
+Although all Active Record tables have an ID column, the URL is the unique identifier and all business logic is tied to it.
+
+If you are new to Ruby, we suggest using rbenv to download the latest MRI Ruby (1.9.3 at the moment) and manage your gems.
+
+Please write tests for any areas lacking them and certainly for any new code.  We are using Test::Unit.

data/README.md

@@ -0,0 +1,36 @@
+# Flatfish 
+Bottom-feeding fun!
+
+## Description 
+Flatfish is a lib to scrape HTML based on a CSV w/ CSS selectors and configurable attributes (eg, page titles).
+The ultimate goal of Flatfish is to prep and load the HTML into Drupal.
+
+## INSTALLATION
+Flatfish is still in development, so it's not on Rubygems just yet.  You'll need to build and install the gem manually, this is really pretty easy.  Assuming you're starting from scratch:
+
+1. We're using Ruby 1.9.3, so install that with RVM, rbenv+ruby-build, or on your own.
+2. Flatfish has a few dependencies, which are listed in the Gemfile, you can install the bundler gem and then use it to grab the rest of the gems at the versions specified in the Gemfile.lock--this is probably a good idea.  The gems can also be installed by hand--there are only a few.
+3. We've set up a quick Rake task to build and install the Flatfish gem, so if you're using RVM (system-wide flavor) just run 'rake install\_gem'.  Otherwise, you can just 'gem build flatfish.gemspec' and 'gem install flatfish-VERSION.gem' according to your setup.
+
+## NOTES
+As Flatfish scrapes the HTML over-the-wire, it can be a bit slow (say 10 minutes for 500 pages), you can speed things up by pointing to a local copy of your site.
+
+## USAGE INSTRUCTIONS
+1. Create a MySQL database
+2. Make a directory for you CSV and configuration file
+3. Create CSVs of URLs w/ CSS selectors (see the example directory), one for each Drupal Content Type
+4. Configure your yaml w/ project specifics (see Pleuronectiformes.yaml as a sample or the example directory)
+5. Run 'flatfish' in your project directory (with the CSV and the config file)
+6. Additional Flatfish runs will update (IE, overwrite) database content based on URL.
+
+## License
+
+(The MIT License)
+
+Copyright (c) 2012 Tim Loudon
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,23 @@
+require 'rake'
+require 'rake/testtask'
+
+task :default => [:test_units]
+
+desc "Run basic tests"
+Rake::TestTask.new("test_units") do |t|
+  t.pattern = 'test/*_test.rb'
+  t.verbose = false 
+  t.warning = true
+end
+
+RUBY='1.9.3'
+
+desc "Build gem"
+task :build_gem do
+  system "rvm #{RUBY} do gem build flatfish.gemspec"
+end
+
+desc "Install gem"
+task :install_gem => :build_gem do
+  system "sudo rvm #{RUBY} do gem install flatfish-*.gem"
+end

data/TODO.md

@@ -0,0 +1,11 @@
+#TODOS
+
+##Add tests
+need more tests, current test coverage is weak and only covers absolutification functionality
+
+##Restructure the code
+need better overall design and refactor the dupe blocks for images and hrefs
+
+##Add fork
+
+##Add logging

data/bin/flatfish

@@ -0,0 +1,48 @@
+#!/usr/bin/env ruby
+
+require 'flatfish'
+require 'getoptlong'
+
+def banner error_code
+  puts "Flatfish scrapes HTML\n\n"
+  puts "Run flatfish in a directory with a config.yml file or pass one in"
+  puts "For more information see the README or visit the repo https://github.com/drupalstaffing/flatfish\n\n"
+  puts "Usage: flatfish myconfig.yml OR flatfish"
+  puts "Options:"
+  puts "-h|--help"
+  puts "-v|--version"
+  exit error_code
+end
+
+opts = GetoptLong.new(
+  [ "--help",        "-h",   GetoptLong::NO_ARGUMENT ],
+  [ "--version",     "-v",   GetoptLong::NO_ARGUMENT ]
+)
+
+begin
+  opts.each do |opt, arg|
+    case opt
+    when "--help"
+      banner 0 
+
+    when "--version"
+      puts "VERSION " + Flatfish::VERSION
+      exit 0
+    end
+  end
+
+rescue 
+  banner 1 
+end
+
+if !File.exists?('config.yml') && ARGV.empty? then
+  puts "ERROR: You need a configuration file.\n\n" 
+  banner 1
+end
+
+puts "Running Flatfish..."
+puts Time.now
+plueronectiforme = Flatfish.new ARGV[0]
+plueronectiforme.ingurgitate
+puts "All done"
+puts Time.now

data/example/article.csv

@@ -0,0 +1,13 @@
+url,path,title,body
+http://drupalconnect.com/blog/e-commerce-problems-solutions-and-considerations,blog/ecommerce-overview,Ecommerce Overview,.region-content-inner
+http://drupalconnect.com/blog/mobile-design-trends-5-top-tips-kathy-chavez-senior-themer,blog/mobile-tips,Mobile Design Tips,.region-content-inner
+http://drupalconnect.com/blog/drupal-migration-migrate-module-best-practices,blog/migration-best-practices,Drupal Migration Best-Practices,.region-content-inner
+http://drupalconnect.com/blog/interview-roger-soper-senior-developer-drupal-support,blog/interview-soper,Roger Soper Interview,.region-content-inner
+http://drupalconnect.com/blog/interview-john-florez-ceo-drupal-connect,blog/interview-florez,John Florez Interview,.region-content-inner
+http://drupalconnect.com/blog/interview-chris-boag-vp-professional-services,blog/interview-boag,Chris Boag Interview,.region-content-inner
+http://drupalconnect.com/blog/responsive-web-design,blog/responsive-design,Responsive Design Overview,.region-content-inner
+http://drupalconnect.com/blog/how-learn-drupal,blog/learning-drupal-tips,Learning Drupal Tips,.region-content-inner
+http://drupalconnect.com/blog/replacing-drupal-static-navigation-apache-solr-facets,blog/solr-navigation,Using Apache for Navigation,.region-content-inner
+http://drupalconnect.com/blog/creating-drupal-context-plugins,blog/context-plugins,Creating Context Plugins,.region-content-inner
+http://drupalconnect.com/blog/stress-testing-your-servers-maxclients-directive,blog/stress-testing-apache,Stress Testing Apache,.region-content-inner
+http://drupalconnect.com/blog/drupal-check-ubercart-check,blog/ubercart-tips,Ubercart Tips,.region-content-inner

data/example/config.yml

@@ -0,0 +1,17 @@
+local_source: '' #use the web
+
+db_user: 'root'
+db_pass: 'root'
+db: 'flatfish_sample'
+
+# NOTE: these map to Drupal content types and AR database tables
+types:
+  Article:
+    csv: '/home/tloudon/workspace/flatfish/example/article.csv'
+    host: 'http://drupalconnect.com'
+  Page:
+    csv: '/home/tloudon/workspace/flatfish/example/page.csv'
+    host: 'http://drupalconnect.com'
+
+development:
+  max_rows: 1000

data/example/page.csv

@@ -0,0 +1,8 @@
+url,path,title,body,field_second_value
+http://drupalconnect.com,front,Drupal Connect,#block-block-13|#block-views-portfolio-block-1,
+http://drupalconnect.com/services,services,Drupal Services,.views-row-first.highlight-box,.views-row-last.highlight-box
+http://drupalconnect.com/services/drupal-data-migration,services/migration,Drupal Migration,.region-content-inner,
+http://drupalconnect.com/services/drupal-development,services/drupal-development,Drupal Development,.region-content-inner,
+http://drupalconnect.com/projects/smosh,portfolio/smosh,Drupal Connect Portfolio: Smosh,.region-content-inner,
+http://drupalconnect.com/projects/greenopoliscom,portfolio/thinkgreenrewards,Drupal Connect Portfolio: Think Green Rewards,.region-content-inner,
+http://drupalconnect.com/projects/stanford-university,portfolio/stanford-school-engineering,Drupal Connect Portfolio: Stanford,.region-content-inner,

data/flatfish.gemspec

@@ -0,0 +1,19 @@
+# -*- encoding: utf-8 -*-
+$:.push File.expand_path("../lib", __FILE__)
+require "flatfish"
+
+Gem::Specification.new do |s|
+  s.name        = 'flatfish'
+  s.version     = Flatfish::VERSION
+  s.date        = '2011-12-28'
+  s.summary     = "Scrape web pages!"
+  s.description = "flatfish accepts a CSV of URLS with CSS selectors prepping them for insert into drupal"
+  s.authors     = ["Tim Loudon", "Mike Crittenden"]
+  s.email       = 'timl@drupalconnect.com'
+  s.homepage    = 'https://github.com/drupalstaffing/flatfish'
+
+  s.files         = `git ls-files`.split("\n")
+  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  s.require_paths = ["lib"]
+end

data/lib/flatfish.rb

@@ -0,0 +1,37 @@
+require 'rubygems'
+require 'yaml'
+require 'csv'
+require 'open-uri'
+require 'uri'
+require 'nokogiri'
+require 'fileutils'
+require 'active_record'
+
+require 'awesome_print'
+
+require 'flatfish/pleuronectiformes'
+require 'flatfish/page'
+require 'flatfish/media'
+require 'flatfish/create_tables'
+require 'flatfish/url'
+
+#API
+module Flatfish 
+  class << self
+    #allow alias Flatfish.new etc
+    def new(yml = nil)
+      yml ||= './config.yml'
+      if !File.exists?(yml) then
+        puts "ERROR: No Config file"
+        exit 1
+      end
+      Flatfish::Pleuronectiformes.new(yml)
+    end
+
+    def ingurgitate
+      new.ingurgitate
+    end
+  end
+
+  VERSION = "0.3.1"
+end

data/lib/flatfish/create_tables.rb

@@ -0,0 +1,29 @@
+module Flatfish 
+
+  class CreateKlass < ActiveRecord::Migration
+    # assume every klass has a URL, Path, Title
+    # pass in additional columns from CSV
+    def self.setup(schema, klass)
+      k = klass.tableize.to_sym
+      create_table(k) do |t|
+        t.string :url
+        t.string :path
+        t.string :title
+      end
+      schema.each do |column|
+        add_column(k, column.gsub(/\s+/, '_').downcase.to_sym, :text)
+      end
+    end
+  end
+
+  class CreateMedia < ActiveRecord::Migration
+    #create media table
+    def self.setup
+      create_table :media do |t|
+        t.string :url
+        t.binary :contents, :limit => 4294967295
+      end
+    end
+  end
+
+end

data/lib/flatfish/media.rb

@@ -0,0 +1,5 @@
+module Flatfish 
+  class Media < ActiveRecord::Base
+    attr_reader :url, :contents
+  end
+end

data/lib/flatfish/page.rb

@@ -0,0 +1,148 @@
+# -*- coding: utf-8 -*- #specify UTF-8 (unicode) characters
+require_relative 'url'
+
+module Flatfish 
+
+  class Page < ActiveRecord::Base
+    self.abstract_class = true
+    @columns = []
+    extend Flatfish::Url
+
+    attr_reader :url, :data
+    attr_accessor :cd
+
+    # Setup - unpack the vars for the web page to be scraped
+    #
+    # csv - an array w/ all of the page specific
+    # config - has some key deets, where to save images, etc that the page has to know
+    # schema - dynamic column headers
+    def setup(csv, config, schema, host)
+      #parse the csv
+      @url, @path, @title  = csv[0], csv[1], csv[2]
+      @fields = []
+      csv[3..-1].each do |field|
+        unless field.nil?
+          @fields << (field.strip! || field)
+        else
+          @fields << -1 #flag
+        end
+      end
+
+      #current directory, we want http://example.com/about/ or http://example.com/home/
+      @cd = (@url[-1,1] == '/')? @url: @url.slice(0..@url.rindex('/'))
+      @schema = schema
+      @host = host
+      @local_source = config['local_source']
+
+      # handle url == host, fix mangled @cd
+      if @url == @host
+        @cd = @url + '/'
+      end
+      Flatfish::Url.creds = {:http_basic_authentication => [config['basic_auth_user'], config['basic_auth_pass']]}
+    end
+
+    def process
+      load_html
+      self.attributes = prep
+    end
+
+    # load html from local or web
+    def load_html
+      file = @local_source + @url.sub(@host, '')
+      if (@url != @host) && !@local_source.nil? && File.exists?(file)
+        f = File.open(file)
+        @doc = Nokogiri::XML(f)
+        f.close
+      else
+        html = Flatfish::Url.open_url(@url)
+        @doc = Nokogiri::HTML(html)
+      end
+    end
+
+    def prep
+      #default to csv, fallback to title element
+      @title = @title.nil? ? @doc.title: @title
+
+      #build a hash of field => data
+      html = Hash.new
+      @fields.each_with_index do |selectors, i|
+        next if -1 == selectors
+        html[@schema[i]] = ''
+        selectors.split('|').each do |selector|
+          update_hrefs(selector)
+          update_imgs(selector)
+          if @doc.css(selector).nil? then
+            field = ''
+          else
+            # sub tokens and gnarly MS Quotes
+            field = @doc.css(selector).to_s.gsub("%5BFLATFISH", '[').gsub("FLATFISH%5D", ']').gsub(/[”“]/, '"').gsub(/[‘’]/, "'")
+          end
+          html[@schema[i]] +=  field
+        end
+      end
+      @data = {
+        'url' => @url,
+        'title' => @title,
+        'path' => @path
+      }
+      @data.merge!(html)
+    end
+
+    # processes link tags
+    # absolutifies and passes media links on for tokenization
+    def update_hrefs(css_selector)
+      @doc.css(css_selector + ' a').each do |a|
+
+        #TODO finalize list of supported file types 
+        href = Flatfish::Url.absolutify(a['href'], @cd)
+        valid_exts = ['.doc', '.docx', '.pdf', '.pptx', '.ppt', '.xls', '.xlsx']
+        if href =~ /#{@host}/  && valid_exts.include?(File.extname(href))
+          media = get_media(href)
+          href = "[FLATFISHmedia:#{media.id}FLATFISH]"
+        end
+        a['href'] = href
+      end
+    end
+
+    # processes image tags
+    # absolutifies images and passes internal ones on for tokenization
+    def update_imgs(css_selector)
+      @doc.css(css_selector + ' img').each do |img|
+        next if img['src'].nil?
+
+        # absolutify and tokenize our images
+        src = Flatfish::Url.absolutify(img['src'], @cd)
+        if src =~ /#{@host}/
+          # check to see if it already exists
+          media = get_media(src)
+          img['src'] = "[FLATFISHmedia:#{media.id}FLATFISH]"
+        end
+      end
+    end
+
+    #TODO replace w/ find_or_create
+    def get_media(url)
+      media = Flatfish::Media.find_by_url(url)
+      if media.nil?
+        media = Flatfish::Media.create(:url => url) do |m|
+          m.contents = read_in_blob(url)
+        end
+      end
+      media
+    end 
+
+    # read in blob
+    def read_in_blob(url)
+      # assume local file
+      file = url.sub(@host, @local_source)
+
+      unless @local_source.nil? || !File.exists?(file)
+        blob = file.read
+      else
+        blob = Flatfish::Url.open_url(URI.escape(url))
+      end
+      blob
+    end
+
+  end
+end

data/lib/flatfish/pleuronectiformes.rb

@@ -0,0 +1,123 @@
+require_relative 'page'
+
+module Flatfish
+
+  class Pleuronectiformes
+    attr_reader :config, :schema, :klasses
+
+    # load in the config
+    def initialize(ymal)
+      @config = YAML.load_file(ymal)
+      db_conn() # establish AR conn
+      @klasses = Hash.new
+    end
+
+    # main loop for flatfish
+    def ingurgitate
+      create_media unless Flatfish::Media.table_exists?
+
+      @config["types"].each do |k,v|
+        next if v["csv"].nil?
+        @csv_file = v["csv"]
+        @host = v["host"]
+        create_klass(k)
+        parse(k)
+      end
+      output_schema
+    end
+
+    # Create the Klass
+    # create table if necessary: table must exist!
+    # create dynamic model
+    def create_klass(k)
+        # commence hackery
+        create_table(k) unless ActiveRecord::Base.connection.tables.include?(k.tableize)
+        @klass = Class.new(Page)
+        @klasses[k] = @klass
+        @klass.table_name = k.tableize
+    end
+
+    def create_table(klass)
+      load_csv
+      Flatfish::CreateKlass.setup(@schema, klass)
+    end
+
+    def create_media
+      Flatfish::CreateMedia.setup
+    end
+
+    #load csv, set schema
+    def load_csv
+      csv = CSV.read(@csv_file)
+      @schema = csv.shift[3..-1]
+      return csv
+    end
+
+    # loop thru csv
+    def parse(k)
+      csv = load_csv
+      @cnt = 0
+      csv.each do |row|
+        begin
+          break if @cnt == @config['max_rows']
+          @cnt += 1
+          page = @klass.find_or_create_by_url(row[0])
+          puts "Processing #{k}.#{page.id} with URL #{row[0]}"
+          page.setup(row, @config, @schema, @host)
+          page.process
+          page.save!
+        rescue Exception => e
+          if e.message =~ /(redirection forbidden|404 Not Found)/
+            ap "URL: #{page.url} #{e}"
+          else
+            ap "URL: #{page.url} ERROR: #{e} BACKTRACE: #{e.backtrace}"
+          end
+        end
+      end
+    end
+
+
+    # generate a dynamic schema.yml for Migrate mapping
+    def output_schema
+      # TODO REFACTOR THIS ISH
+      klasses = @klasses
+      File.open('schema.yml', 'w') do |out|
+        output = Hash.new
+        output["schema"] = Hash.new
+        klasses["Media"] = Flatfish::Media
+
+        klasses.each_pair do |k,v|
+          klass_attributes = Hash.new
+          v.new.attributes.each { |a| klass_attributes[a[0]] = split_type(v.columns_hash[a[0]].sql_type) }
+          output["schema"].merge!({k => {"machine_name" => k.tableize, "fields" => klass_attributes, "primary key" => ["id"]}})
+        end
+        out.write output.to_yaml
+      end
+    end
+
+    # helper function to convert AR sql_type to
+    # Drupal format;
+    # eg :type => varchar(255) to :type => varchar, :length => 255
+    def split_type type
+      if type =~ /\(/ then
+        x = type.split("(")
+        return {"type" => x[0], "length" => x[1].sub(")","").to_i }
+      else
+        return {"type" => type}
+      end
+    end
+
+
+    def db_conn
+      ActiveRecord::Base.establish_connection(
+                                        :adapter=> "mysql2",
+                                        :host => "localhost",
+                                        :username => @config['db_user'],
+                                        :password => @config['db_pass'],
+                                        :database=> @config['db']
+                                       )
+    end
+
+  end
+
+end

data/lib/flatfish/url.rb

@@ -0,0 +1,57 @@
+module Flatfish 
+  module Url
+    #methods for handling URLs
+    class << self
+      attr_accessor :creds
+
+
+      # Handle SSL Redirects + HTTP Auth
+      # to catch linked files @ runtime
+      def open_url url
+        begin
+          html = open(url).read
+        rescue Exception => e
+          redirect = URI.parse(url)
+          if e.message =~ /redirection forbidden/ && redirect.scheme == 'http'
+            html = open_url("https://" + redirect.host + redirect.path)
+          end
+          if e.message =~ /(Authorization Required|Unauthorized)/
+            html = open(url, @creds).read
+          end
+        end
+        return html
+      end
+
+      # take a URL, return an absolute URL
+      def absolutify url, cd 
+        url = url.to_s
+        # deal w/ bad URLs, already absolute, etc
+        begin
+          u = URI.parse(url)
+        rescue
+          # GIGO, no need for alarm
+          return url
+        end
+
+        return url if u.absolute? # http://example.com/about
+        c = URI.parse(cd)
+        return c.scheme + "://" + c.host + url if url.index('/') == 0 # /about
+        return cd + url if url.match(/^[a-zA-Z]+/) # about*
+
+        # only relative from here on in; ../about, ./about, ../../about
+        u_dirs = u.path.split('/')
+        c_dirs = c.path.split('/')
+
+        # move up the directory until there are no more relative paths
+        u.path.split('/').each do |x|
+          break unless (x == '' || x == '..' || x == '.')
+          u_dirs.shift
+          c_dirs.pop unless x == '.'
+        end
+        return c.scheme + "://" + c.host + c_dirs.join('/') + '/' +  u_dirs.join('/')
+      end
+
+    end
+  end
+
+end

data/test/url_test.rb

@@ -0,0 +1,37 @@
+$:.push File.expand_path("../../lib", __FILE__)
+require 'flatfish/url'
+require 'uri'
+require 'test/unit'
+
+class URL_Tests < Test::Unit::TestCase
+
+  def setup
+    @cd = "http://example.com/test_dir1/test_dir2/"
+  end
+
+  def test_relative
+    tst = Flatfish::Url.absolutify("../images/test.png", @cd)
+    assert_equal("http://example.com/test_dir1/images/test.png", tst)
+  end
+
+  def test_double_relative
+    tst = Flatfish::Url.absolutify("../../images/test.png", @cd)
+    assert_equal("http://example.com/images/test.png", tst)
+  end
+
+  def test_absolute
+    tst = Flatfish::Url.absolutify("http://example.com/images/test.png", @cd)
+    assert_equal("http://example.com/images/test.png", tst)
+  end
+
+  def test_broken_root
+    tst = Flatfish::Url.absolutify("../../../../images/test.png", @cd)
+    assert_equal("http://example.com/images/test.png", tst)
+  end
+
+  def test_same_dir
+    tst = Flatfish::Url.absolutify("test.png", @cd)
+    assert_equal("http://example.com/test_dir1/test_dir2/test.png", tst)
+  end
+
+end

metadata

@@ -0,0 +1,66 @@
+--- !ruby/object:Gem::Specification
+name: flatfish
+version: !ruby/object:Gem::Version
+  version: 0.3.1
+  prerelease: 
+platform: ruby
+authors:
+- Tim Loudon
+- Mike Crittenden
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2011-12-28 00:00:00.000000000 Z
+dependencies: []
+description: flatfish accepts a CSV of URLS with CSS selectors prepping them for insert
+  into drupal
+email: timl@drupalconnect.com
+executables:
+- flatfish
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- Gemfile
+- Gemfile.lock
+- HACKING.md
+- README.md
+- Rakefile
+- TODO.md
+- bin/flatfish
+- example/article.csv
+- example/config.yml
+- example/page.csv
+- flatfish.gemspec
+- lib/flatfish.rb
+- lib/flatfish/create_tables.rb
+- lib/flatfish/media.rb
+- lib/flatfish/page.rb
+- lib/flatfish/pleuronectiformes.rb
+- lib/flatfish/url.rb
+- test/url_test.rb
+homepage: https://github.com/drupalstaffing/flatfish
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.24
+signing_key: 
+specification_version: 3
+summary: Scrape web pages!
+test_files: []