agio 0.5.0

data/.rspec

@@ -0,0 +1,2 @@
+--colour
+--format documentation

data/History.rdoc

@@ -0,0 +1,3 @@
+=== 0.5 / 2011-09-20
+
+* Initial release for public consumption.

data/License.rdoc

@@ -0,0 +1,23 @@
+== License
+
+This software is available under the terms of the MIT license.
+
+* Copyright 2011 Austin Ziegler
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
+of the Software, and to permit persons to whom the Software is furnished to do
+so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/Manifest.txt

@@ -0,0 +1,23 @@
+.rspec
+History.rdoc
+License.rdoc
+Manifest.txt
+README.rdoc
+Rakefile
+agio.gemspec
+bin/agio
+lib/agio.rb
+lib/agio/block.rb
+lib/agio/bourse.rb
+lib/agio/broker.rb
+lib/agio/data.rb
+lib/agio/flags.rb
+lib/agio/html_element_description.rb
+spec/block_spec.rb
+spec/bourse_spec.rb
+spec/broker_spec.rb
+spec/data_spec.rb
+spec/flags_spec.rb
+spec/html_element_description_spec.rb
+spec/pmh_spec.rb
+spec/spec_helper.rb

data/README.rdoc

@@ -0,0 +1,97 @@
+= Agio
+
+== Description
+Agio is a library and a tool for converting HTML to
+{Markdown}[http://daringfireball.net/projects/markdown/].
+
+=== About the Name
+
+The name was chosen because <em>agio</em> is "a premium on money in exchange",
+sort of the opposite of a markdown. It comes from the Italian <em>aggio</em>
+(premium), not from the Italian <em>agio</em> (ease), although the hope is that
+there is an ease in use of this library.
+
+== Why Agio?
+
+1. Agio is well tested.
+2. Agio is pure Ruby.
+3. Agio is MIT licensed.
+
+=== Agio is Well Tested
+
+With this release, there are 274 unit tests (RSpec examples) covering almost
+everything. The only things not covered are Agio itself (which mostly performs
+coordination duties) and Agio::Bourse (the parsed HTML-to-Markdown translator).
+These tests cover 85%–95% of the code known to be tested, and the modules
+currently missing test will have those tests completed prior to the release of
+Agio 1.0.
+
+In addition to the unit tests, more than a hundred manual tests have been run
+to verify the quality of output for basic HTML. These manual tests have taken
+one of two forms:
+
+1. Markdown input converted to HTML with
+   {rdiscount}[https://github.com/rtomayko/rdiscount], and then converted back
+   to Markdown with Agio. In all cases, the resulting Markdown is either
+   identical to the original or the differences can be attributed to style
+   (how Agio writes emphasized text versus the hand-written original, or how
+   Agio represents links by default). This tests Agio's round-trip capability.
+2. HTML input converted to Markdown with either Pandoc or html2txt.py,
+   converted <em>back</em> to HTML with rdiscount, and then converted once
+   again to Markdown with Agio. This tests Agio's ability to output data
+   that is syntactically similar to those of better-known and presumably
+   better-tested tools.
+
+Agio will likely have bugs, especially before version 1.0, and not all features
+are yet implemented or exposed to the user. Syntactic support is also
+incomplete, as the goal is to support many of the syntax extensions found in
+{Markdown Extra}[http://michelf.com/projects/php-markdown/extra/] or other
+popular modules, such as Github-flavoured Markdown.
+
+== Where
+
+* {GitHub}[https://github.com/halostatue/agio]
+
+== Synopsis
+
+Install Agio with:
+
+  gem install agio
+
+Run Agio against HTML with:
+
+  agio input.html > output.markdown
+
+== History
+
+=== Why I Wrote Agio
+
+Agio is the result of some yak-shaving as I was looking to convert my blog
+content from {WordPress}[http://wordpress.org] to
+{Jekyll}[https://github.com/mojombo/jekyll]. The Jekyll wiki points to Thomas
+Frőssman's {Exitwp}[https://github.com/thomasf/exitwp] Python script as a
+reliable conversion mechanism, but I found that it couldn't handle the data in
+my {WordPress export file}[http://codex.wordpress.org/Tools_Export_Screen]. So,
+I ported Exitwp from Python to Ruby as
+{Poole}[https://github.com/halostatue/poole].
+
+Like Exitwp, Poole depends on {Pandoc}[http://johnmacfarlane.net/pandoc/].
+While it's an amazing tool, it took the better part of 45 minutes to compile
+the {Haskell Platform}[http://hackage.haskell.org/platform/] with
+{Homebrew}[https://github.com/mxcl/homebrew] and Pandoc with
+{Cabal}[http://www.haskell.org/cabal/]. Looking around the Ruby community,
+there wasn't a single Ruby-based HTML-to-Markdown converter that I felt I could
+trust to get everything right that was also licensed to my liking (I prefer the
+MIT license). While {Kramdown}[http://kramdown.rubyforge.org/] is impressive,
+it's GPL-licensed. I didn't want Poole (which is MIT-licensed) to depend on
+anything that provided any less freedom for any purpose.
+
+Armed with this plan, I started the process of deeply understanding how Aaron
+Swartz's {html2txt.py}[http://www.aaronsw.com/2002/html2text/] script works.
+This included an early version of Agio that was a more-or-less straightforward
+port, but produced output that was worse because of differences between
+Python's +textwrap+ module and Ruby's
+{Text::Format}[http://rubyforge.org/projects/text-format] that could not be
+cleanly resolved by tweaking Aaron's algorithm.
+
+:include: License.rdoc

data/Rakefile

@@ -0,0 +1,35 @@
+# -*- ruby encoding: utf-8 -*-
+
+require 'rubygems'
+require 'rspec'
+require 'hoe'
+
+Hoe.plugin :doofus
+Hoe.plugin :gemspec
+Hoe.plugin :git
+
+Hoe.plugins.delete :rcov
+
+spec = Hoe.spec 'agio' do
+  developer('Austin Ziegler', 'austin@rubyforge.org')
+
+  self.history_file = 'History.rdoc'
+  self.readme_file = 'README.rdoc'
+  self.extra_rdoc_files = FileList["*.rdoc"].to_a
+
+  self.extra_deps << ['nokogiri', '~> 1.5.0']
+  self.extra_deps << ['main', '~> 4.7.3']
+  self.extra_dev_deps << ['rspec', '~> 2.0']
+  self.extra_dev_deps << ['hoe-doofus', '~> 1.0']
+  self.extra_dev_deps << ['hoe-gemspec', '~> 1.0']
+  self.extra_dev_deps << ['hoe-git', '~> 1.0']
+  self.extra_dev_deps << ['hoe-seattlerb', '~> 1.0']
+end
+
+RSpec::Core::RakeTask.new(:rcov) do |t|
+  t.rcov = true
+  t.rcov_opts =  %q[--exclude "osx/objc,gems/,spec/,features/"]
+  t.verbose = true
+end
+  
+# vim: syntax=ruby

data/agio.gemspec

@@ -0,0 +1,53 @@
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+  s.name = "agio"
+  s.version = "0.5.0"
+
+  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+  s.authors = ["Austin Ziegler"]
+  s.date = "2011-09-20"
+  s.description = "Agio is a library and a tool for converting HTML to\n{Markdown}[http://daringfireball.net/projects/markdown/]."
+  s.email = ["austin@rubyforge.org"]
+  s.executables = ["agio"]
+  s.extra_rdoc_files = ["Manifest.txt", "History.rdoc", "License.rdoc", "README.rdoc"]
+  s.files = [".rspec", "History.rdoc", "License.rdoc", "Manifest.txt", "README.rdoc", "Rakefile", "agio.gemspec", "bin/agio", "lib/agio.rb", "lib/agio/block.rb", "lib/agio/bourse.rb", "lib/agio/broker.rb", "lib/agio/data.rb", "lib/agio/flags.rb", "lib/agio/html_element_description.rb", "spec/block_spec.rb", "spec/bourse_spec.rb", "spec/broker_spec.rb", "spec/data_spec.rb", "spec/flags_spec.rb", "spec/html_element_description_spec.rb", "spec/pmh_spec.rb", "spec/spec_helper.rb", ".gemtest"]
+  s.rdoc_options = ["--main", "README.rdoc"]
+  s.require_paths = ["lib"]
+  s.rubyforge_project = "agio"
+  s.rubygems_version = "1.8.10"
+  s.summary = "Agio is a library and a tool for converting HTML to {Markdown}[http://daringfireball.net/projects/markdown/]."
+
+  if s.respond_to? :specification_version then
+    s.specification_version = 3
+
+    if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
+      s.add_runtime_dependency(%q<nokogiri>, ["~> 1.5.0"])
+      s.add_runtime_dependency(%q<main>, ["~> 4.7.3"])
+      s.add_development_dependency(%q<rspec>, ["~> 2.0"])
+      s.add_development_dependency(%q<hoe-doofus>, ["~> 1.0"])
+      s.add_development_dependency(%q<hoe-gemspec>, ["~> 1.0"])
+      s.add_development_dependency(%q<hoe-git>, ["~> 1.0"])
+      s.add_development_dependency(%q<hoe-seattlerb>, ["~> 1.0"])
+      s.add_development_dependency(%q<hoe>, ["~> 2.12"])
+    else
+      s.add_dependency(%q<nokogiri>, ["~> 1.5.0"])
+      s.add_dependency(%q<main>, ["~> 4.7.3"])
+      s.add_dependency(%q<rspec>, ["~> 2.0"])
+      s.add_dependency(%q<hoe-doofus>, ["~> 1.0"])
+      s.add_dependency(%q<hoe-gemspec>, ["~> 1.0"])
+      s.add_dependency(%q<hoe-git>, ["~> 1.0"])
+      s.add_dependency(%q<hoe-seattlerb>, ["~> 1.0"])
+      s.add_dependency(%q<hoe>, ["~> 2.12"])
+    end
+  else
+    s.add_dependency(%q<nokogiri>, ["~> 1.5.0"])
+    s.add_dependency(%q<main>, ["~> 4.7.3"])
+    s.add_dependency(%q<rspec>, ["~> 2.0"])
+    s.add_dependency(%q<hoe-doofus>, ["~> 1.0"])
+    s.add_dependency(%q<hoe-gemspec>, ["~> 1.0"])
+    s.add_dependency(%q<hoe-git>, ["~> 1.0"])
+    s.add_dependency(%q<hoe-seattlerb>, ["~> 1.0"])
+    s.add_dependency(%q<hoe>, ["~> 2.12"])
+  end
+end

data/bin/agio

@@ -0,0 +1,17 @@
+#!ruby
+
+# main 4.7.3 has a warning in main.rb:
+# main-4.7.3/lib/main.rb:28: warning: redefine libdir
+#
+# map 4.3.0 has a warning in map.rb:
+# map-4.3.0/lib/map.rb:551: warning: method redefined; discarding old to_map
+#
+# Therefore, no warnings.
+#!ruby -w
+
+require 'agio'
+
+ARGV.each do |arg|
+  html = File.open(arg) { |f| f.read }
+  puts Agio.to_markdown(html)
+end

data/lib/agio.rb

@@ -0,0 +1,171 @@
+# -*- ruby encoding: utf-8 -*-
+
+require 'nokogiri'
+require 'stringio'
+require 'text/format'
+
+##
+# = Agio
+# Agio converts HTML to Markdown.
+#
+# == About the Name
+# The name was chosen because agio is "a premium on money in exchange",
+# sort of the opposite of a markdown. It comes from the Italian aggio
+# (premium), not from the Italian agio (ease), although the hope is that
+# there is an ease in use of this library.
+#
+# It is structurally based on Aaron Swarz's html2txt Python script inasmuch
+# as the SAX parsing he does is also done here and with pretty much the same
+# behaviour.
+#
+# == License
+# This code is licensed under MIT License
+class Agio
+  VERSION = "0.5.0"
+end
+
+require 'agio/flags'
+require 'agio/broker'
+require 'agio/bourse'
+
+class Agio
+  ##
+  # :attr_accessor:
+  # The default HTML document to be processed. Because the #parse method can
+  # be called with an HTML document, this does not *need* to be set.
+  attr_accessor :html
+
+  ##
+  # :attr_reader:
+  # The width of the body text for the generated Markdown text outside of
+  # +pre+ bodies and other items which do not wrap well in most Markdown
+  # parsers.
+  def columns
+    bourse.formatter.columns
+  end
+  ##
+  # :attr_writer:
+  # The width of the body text for the generated Markdown text outside of
+  # +pre+ bodies and other items which do not wrap well in most Markdown
+  # parsers.
+  #
+  # If +nil+ is provided, the default value of 78 is set.
+  def columns=(value)
+    bourse.formatter.columns = value || 78
+    bourse.formatter.columns
+  end
+
+  ##
+  # :attr_reader:
+  # Controls how links are placed in the Markdown document.
+  def link_placement
+    bourse.link_placement
+  end
+  # :attr_writer: link_placement
+  # Controls how links are placed in the Markdown document.
+  #
+  # In-Line::   Links appear next to their wrapped text, like "[See the
+  #             example](http://example.org/)". The default for
+  #             link_placement, set if the value is +nil+, <tt>:inline</tt>,
+  #             or any other unrecognized value.
+  # Paragraph:: Links appear in the body as linked references, like "[See
+  #             the example][1]", and the reference "[1]:
+  #             http://example.org" is placed immediately after the
+  #             paragraph in which the link first appears. Used if the value
+  #             of link_placement is <tt>:paragraph</tt>.
+  # Endnote::   Links appear in the body as linked references, like "[See
+  #             the example][1]", and the reference "[1]:
+  #             http://example.org" is placed at the end of the document.
+  #             Used if the value of link_placement is <tt>:endnote</tt>.
+  def link_placement=(value)
+    bourse.link_placement = value
+  end
+
+  ##
+  # :attr_reader: base_url
+  # The base URL for implicit (or local) link references. If not provided,
+  # links will remain implicit. This is a String value.
+  def base_url
+    bourse.base_url
+  end
+  ##
+  # :attr_writer: base_url
+  # The base URL for implicit (or local) link references. If not provided,
+  # links will remain implicit. This is a String value.
+  def base_url=(value)
+    bourse.base_url = value
+  end
+
+  ##
+  # :attr_reader: skip_local_fragments
+  # Controls whether local link references containing fragments will be
+  # output in the final document.
+  #
+  # A local link reference is either an implicit link reference (one missing
+  # the protocol and host, such as '&lt;a href="about.html"&gt;' or '&lt;a
+  # href="/about.html"&gt;') or one that points to the #base_url.
+  #
+  # If this value is +true+, links that refer to fragments on local URIs
+  # will be ignored (such as '&lt;a href="about.html#address"&gt;').
+  def skip_local_fragments
+    bourse.skip_local_fragments
+  end
+
+  ##
+  # :attr_writer: skip_local_fragments
+  # Controls whether local link references containing fragments will be
+  # output in the final document.
+  #
+  # A local link reference is either an implicit link reference (one missing
+  # the protocol and host, such as '&lt;a href="about.html"&gt;' or '&lt;a
+  # href="/about.html"&gt;') or one that points to the #base_url.
+  #
+  # If this value is +true+, links that refer to fragments on local URIs
+  # will be ignored (such as '&lt;a href="about.html#address"&gt;').
+  def skip_local_fragments=(value)
+    bourse.skip_local_fragments = value
+  end
+
+  def initialize(options = {})
+    @broker = Agio::Broker.new
+    @bourse = Agio::Bourse.new(broker, self)
+
+    self.html           = options[:html]
+    self.columns        = options[:columns]
+    self.link_placement = options[:link_placement]
+
+    yield self if block_given?
+
+    @parser = Nokogiri::HTML::SAX::Parser.new(broker)
+  end
+
+  def parse(html)
+    @parser.parse(html)
+    self
+  end
+  private :parse
+
+  def transform(html)
+    parse(html)
+    bourse.transform
+  end
+  private :transform
+
+  def to_s(html = nil)
+    transform(html || self.html)
+    bourse.output.string
+  end
+  alias to_markdown to_s
+
+  def self.to_markdown(html)
+    self.new.to_markdown(html)
+  end
+
+  attr_reader :broker
+  private :broker
+
+  attr_reader :bourse
+  private :bourse
+end
+
+# vim: ft=ruby

data/lib/agio/block.rb

@@ -0,0 +1,132 @@
+# -*- ruby encoding: utf-8 -*-
+
+require 'agio/html_element_description'
+
+##
+# A Block is the fundamental collection for the Broker that is used to
+# then generate the Markdown.
+class Agio::Block
+  def inspect
+    if options.empty?
+      %Q(#<#{self.class} #{name} #{contents.inspect}>)
+    else
+      %Q(#<#{self.class} #{name}(#{options.inspect}) #{contents.inspect}>)
+    end
+  end
+
+  # The name of the element the Block is for.
+  attr_reader :name
+
+  # The options, if provided, for the element.
+  attr_reader :options
+
+  # The contents of the Block.
+  attr_reader :contents
+
+  # The description of the HTML element the Block represents (this will
+  # always be a Nokogiri::HTML::ElementDescription or +nil+).
+  attr_reader :description
+
+  # Create the Block from a tag start.
+  def initialize(name, options = {})
+    @name, @options = name, options
+    @description = Agio::HTMLElementDescription[name]
+    @contents = []
+  end
+
+  # Append the contents provided to the Block.
+  def append(*contents)
+    @contents.push(*contents)
+  end
+
+  # Returns +true+ if the Block is a standard HTML element (as understood
+  # by Nokogiri).
+  def standard?
+    !!description
+  end
+
+  ##
+  # Returns +true+ if this Block is an HTML inline element.
+  def inline?
+    description && description.inline?
+  end
+
+  ##
+  # Returns +true+ if this Block is an HTML block element.
+  def block?
+    description && description.block?
+  end
+
+  ##
+  # Returns +true+ if this Block can contain the other Block provided.
+  def can_contain?(other)
+    case other
+    when Agio::Block
+      description && description.sub_elements.include?(other.name)
+    when String
+      description && description.sub_elements.include?(other)
+    else
+      false
+    end
+  end
+
+  ##
+  # Returns +true+ if the Block is a 'li' (list item) element.
+  def li?
+    name == "li"
+  end
+
+  ##
+  # Returns +true+ if the Block is a 'pre' element.
+  def pre?
+    name == "pre"
+  end
+
+  ##
+  # Returns +true+ if the Block is a 'ul' or 'ol' element.
+  def ul_ol?
+    name == "ul" or name == "ol"
+  end
+
+  ##
+  # Returns +true+ if the Block is a definition item ('dt' or 'dd').
+  def definition?
+    name == "dt" or name == "dd"
+  end
+
+  ##
+  # Returns +true+ if the block is a definition list ('dl') element.
+  def dl?
+    definition? or name == "dl"
+  end
+
+  ##
+  # Determine whether the +other+ Block is a sibling of this Block.
+  # Blocks are siblings if:
+  #
+  # 1. This Block cannot contain the other Block.
+  # 2. This Block is a definition ('dt' or 'dd') and so is the other
+  #    Block.
+  # 3. This Block's name and the other Block's name are the same.
+  def sibling_of?(other)
+    if can_contain? other
+      false
+    elsif definition? and other.definition?
+      true
+    elsif name == other.name
+      true
+    else
+      false
+    end
+  end
+
+  ##
+  # Used mostly for testing.
+  def ==(other)
+    name == other.name &&
+      options == other.options &&
+      contents == other.contents
+  end
+end
+
+# vim: ft=ruby