tartan 0.1.0 → 0.1.1

data/README

@@ -14,7 +14,7 @@ following benefits:
 2. allows layering and extension of parsing rules
 3. allows multiple output formats from the same syntax specification
 
-The current implementation of Tartan is in Ruby and includes a full Markdown
+The current implementation of Tartan is in Ruby and includes a full Markdown[http://daringfireball.net/projects/markdown/]
 parser (described in YAML). The format of the parsing specification has been
 created with an eye to having a language independent definition of wiki (and
 possibly other) mark-ups. That's a lofty goal, and Tartan hasn't quite gotten
@@ -24,9 +24,10 @@ something more than just convert wiki text directly into HTML.
 
 == Usage
 
-So, really all you want to do is generate HTML from Markdown text.  Here's
+So, really all you want to do is generate HTML from Markdown[http://daringfireball.net/projects/markdown/] text.  Here's
 how you do it:
 
+  # require 'rubygems' # if you are pulling Tartan in as a gem
   require 'tartan_markdown'
   
   html = TartanMarkdown.new("* howdy\n* doody").to_html
@@ -43,9 +44,9 @@ called in the same way on the instance of the parser object.
 
 You can add parsing syntax to existing parsers.  This is done by building up a set of parsers specifications that work together.
 
-In the Tartan distribution you have a specification for Markdown and you also
+In the Tartan distribution you have a specification for Markdown[http://daringfireball.net/projects/markdown/] and you also
 have a specification for table mark-up. You can combine them by creating a new
-class that layers the tables onto the Markdown definition as follows in a file
+class that layers the tables onto the Markdown[http://daringfireball.net/projects/markdown/] definition as follows in a file
 called <tt>tartan_markdown_tables.rb</tt>:
 
   require 'tartan_markdown_def'
@@ -58,7 +59,6 @@ called <tt>tartan_markdown_tables.rb</tt>:
   
 In another file you could use this new parser:
 
-
   require 'tartan_markdown_tables'
   
   html = TartanMarkdownTables.new("[|*happy*||**days**|]").to_html
@@ -68,7 +68,7 @@ In another file you could use this new parser:
   
 == The Parsing Specification
 
-
+Each specific parser (Markdown[http://daringfireball.net/projects/markdown/] to HTML, Textile to HTML, your wiki to xml, etc.) needs a parsing specification to tell Tartan how to convert the text into HTML (or what ever other format you need).
 
 === Overall Structure
 
@@ -91,19 +91,19 @@ list of parsing rules. The base context defaults to <tt>block</tt>; that is, the
 The following is a simple parsing rule to match paragraphs and mark them up in HTML:
 
   title: paragraph
-  match: "/(^[^\n]+$\n)+^[^\n]+$/m"
+  match: /(^[^\n]+$\n)*^[^\n]+$/m
   html:
     start_mark: <p>
     end_mark:   </p>
 
 A paragraph, in this case, is any grouping of non blank lines.
 
-The parser will repetitively apply the <tt>match</tt> regular expression and if it matches, the <tt>html</tt> output sub-rule will put <tt><p></tt> and <tt></p></tt> around the text that is matched as a paragraph.
+The parser will repetitively apply the <tt>match</tt> regular expression and if it matches, the <tt>html</tt> output sub-rule will put the <tt>start_mark</tt>, <tt><p></tt>, and the <tt>end_mark</tt>, <tt></p></tt>, around the text that is matched as a paragraph.
 
 If we wanted to also mark off blocks of code that are indented by say 2 or more spaces at the beginning of the line, we could use the following rule:
 
   title: code
-  match: "/(^[ ]{2,}\S.+?$\n)+^[ ]{2,}\S.+?$/m"
+  match: /(^[ ]{2,}\S.+?$\n)+^[ ]{2,}\S.+?$/m
   html:
     start_mark: <pre><code>
     end_mark: </code></pre>
@@ -112,7 +112,7 @@ When we want to add the <tt>code</tt> rule, the ordering becomes important.  If
 
   block:
     - title: code
-      match: "/(^[ ]{2,}\S.+?$\n)+^[ ]{2,}\S.+?$/m"
+      match: /(^[ ]{2,}\S.+?$\n)+^[ ]{2,}\S.+?$/m
       html:
         start_mark: <pre><code>
         end_mark: </code></pre>
@@ -122,9 +122,142 @@ When we want to add the <tt>code</tt> rule, the ordering becomes important.  If
         start_mark: <p>
         end_mark:   </p>
   
+Now, lets say we want to be able to mark-up text with emphasis (HTML <tt><em></tt>) and strong emphasis (HTML <tt><strong></tt>) in paragraph text, but not code.  We'll use an asterisk (*) around text we want to have emphasis and a double asterisk around text we want to have strong emphasis (**).  Note that we don't want this to happen in text in a code block.
+  
+To do this, we set up a new parsing context for paragraph body text and "point" the parser to the context when it recognizes a paragraph.
+
+First, we create the paragraph parsing context:
+
+  paragraph:
+    - title: strong
+      match: /\*\*(.*?)\*\*/
+      html:
+        replace: <strong>\1</strong>
+
+    - rescan
+
+    - title: emphasis
+      match: /\*(.*?)\*/
+      html:
+        replace: <em>\1</em>
+
+The <tt>rescan</tt> directive between the <tt>strong</tt> and <tt>emphasis</tt> rules tells the parser to "start over".  This is needed because otherwise the <tt>strong</tt> rule would "claim" all the text it matched and the <tt>emphasis</tt> rule wouldn't have a chance to parse any of it.  This would come into play if we had a paragraph such as:
+
+  Now listen to this **I want *you* to really hear me**.
+  
+This should get marked up as:
+
+  <p>Now listen to this <strong>I want <em>you<em> to really hear me</strong>.</p>
+  
+but we would get the following without the rescan:
+
+  <p>Now listen to this <strong>I want *you* to really hear me</strong>.</p>
+  
+You might also note that the ordering here, again, is important.  If we leave out the <tt>rescan</tt>, we would get the following output instead:
+
+  <p>Now listen to this <em></em>I want <em>you</em> to really hear me<em></em>.</p>
+
+Now, we also need to modify the paragraph rule in the <tt>block</tt> context to use the new <tt>paragraph</tt> context:
+
+  # . . .
+    - title: paragraph
+      match: /(^[^\n]+$\n)*^[^\n]+$/m
+      subparse: paragraph
+      html:
+        start_mark: <p>
+        end_mark:   </p>
+  # . . .
+
+To do this we use the <tt>subparse</tt> directive to tell the parser that the contents of the paragraph should be parsed by the <tt>paragraph</tt> context.
+
+==== Creating a Mix-in
+
+It's possible to mix-in or layer a parsing specification with a base parser.  This allows you to add additional markup or change the markup of an existing syntax.  You could use this to add table mark-up to Markdown[http://daringfireball.net/projects/markdown/] (in fact, this mix-in to Markdown is available as part of the Tartan code distribution).
+
+To show how this works, we'll look at how to specify and then add character element markup to the parser example we've been working with.  We want to turn things like "<", "&" and "->" into "&lt;", "&amp;" and "&rarr;".
+
+We want these transformation to be done in the context of parsing paragraphs, so we'll only want to add to the <tt>paragraph</tt> context in our previous example.
+
+So, to add this syntax parsing, you would create the following specification:
+
+  paragraph:
+    - rescan
+    - title: amp
+      match: /&/
+      html:
+        replace: '&amp;'
+      rescan: true
+    - title: rightArrow
+      match: /->/
+      html:
+        replace: '&rarr;'
+      rescan: true
+    - title: lessThan
+      match: /</
+      html:
+        replace: '&lt;'
+      rescan: true
+    - title: greaterThan
+      match: />/
+      html:
+        replace: '&gt;'
+
+That's it for the mix-in specification.  Now we add these to the previous set.  We didn't touch on file naming of specifications before, but now we need to.  Let's say that we put the previous specification in a file called <tt>example-parser.yml</tt> and we put the new spec in <tt>entities.yml</tt>.  To combine them, we would create a new Ruby class like this:
+
+  class ExampleParserWithEntities < Tartan
+    yaml "example-parser.yml"
+    yaml "entities.yml"
+  end
+
+By default, the rules of a mix-in are added to the end of any given context.  So, the effective resulting specification once the two sets of rules are combined would be:
+  block:
+    - title: code
+      match: /(^[ ]{2,}\S.+?$\n)+^[ ]{2,}\S.+?$/m
+      html:
+        start_mark: <pre><code>
+        end_mark: </code></pre>
+    - title: paragraph
+      match: /(^[^\n]+$\n)*^[^\n]+$/m
+      subparse: paragraph
+      html:
+        start_mark: <p>
+        end_mark:   </p>
+  paragraph:
+    - title: emphasis
+      match: /\*(.*?)\*/
+      html:
+        replace: <em>\1</em>
+    - rescan
+    - title: amp
+      match: /&/
+      html:
+        replace: '&amp;'
+      rescan: true
+    - title: rightArrow
+      match: /->/
+      html:
+        replace: '&rarr;'
+      rescan: true
+    - title: lessThan
+      match: /</
+      html:
+        replace: '&lt;'
+      rescan: true
+    - title: greaterThan
+      match: />/
+      html:
+        replace: '&gt;'
+
+==== Going Further
+
+Honestly, this brief tutorial just provides you with the basics of Tartan.  If you want to know more, for now, the best thing is to look at the Markdown[http://daringfireball.net/projects/markdown/] and table extension specification in the code.  That will show you a real-world example of how to create a base parser and a mix-in.
+
+There will be additional documentation to follow.  In particular a reference guide that covers all the parser rule directives one at a time.
+
+If you need some help in getting Tartan to work for your project, please don't hesitate to post to the Tartan help-form[http://rubyforge.org/forum/forum.php?forum_id=8042] or write me directly at mailto:bitherder@rubyforge.org.
 
 == The Name
 
 Tartan is intended to weave together different parsing elements.  It's intended
-to be an alternative of both RedCloth[http:www.redcloth.org/] and BlueCloth.  Tartan is a kind of cloth
+to be an alternative of both RedCloth[http:www.redcloth.org/] and BlueCloth[http://www.deveiate.org/projects/BlueCloth].  Tartan is a kind of cloth
 that weaves different colors together in an interesting pattern.

data/lib/markdown.yml

@@ -51,7 +51,8 @@ span:
           (?:span|cite|del|em|strong|dfn|code|samp|kbd|var|abbr
             |acronym|b|i|font|a|img|object|tt|big|small|strike|s|u
             |sup|sub|q)
-          . *?>/x
+          (:?\s+[^\s>][^>]*?)?
+        >/x
     html:
       start_mark: ''
       end_mark: ''
@@ -79,13 +80,14 @@ span:
       replace: '<'
 
   - title: gt
-    match: '/(?:(?!^))>/'
+    match: />/
     html:
       replace: '>'
 
   - title: backslash_escapes
     shelve: true
-    match: /\\([\\\`\*\_\{\}\[\]\(\)\>\#\.\!\+\-])/
+#    match: /\\([\\\`\*\_\{\}\[\]\(\)\>\#\.\!\+\-])/
+    match: /\\([-\[\]\\`*_{}()>#.!+])/   # `
     html:
       replace: '\1'
 
@@ -187,7 +189,7 @@ back_smarties:
       replace: '<' 
 
   - title: gt
-    match: '/(?:(?!^))>/'
+    match: '/>/'
     html:
       replace: '>' 
 
@@ -245,7 +247,7 @@ block:
       end_mark: '</h2>'
 
   - title: hash_header
-    match: '/^(\# {1,6})\s*([^#]*?)(?:\s*\#+\s*|\s*)$/x'
+    match: '/^(#{1,6})\s*([^#\s][^#]*?)(?:\s*\#+)?\s*?$/'
     subparse:
       context: span
       match_group: 2
@@ -301,12 +303,12 @@ block:
       context: listSpaced
       match_group: 3
     html:
-      start_mark: "<ul>\n"
+      start_mark: "\\1\\2<ul>\n"
       end_mark: "\n</ul>" 
 
   - title: bulletListClose
     match: >
-      / (?:\A(\s*)|\n(\n))
+      / (?:\A(\s*)|(\n\n))
                   ([-*+](?:[ ]+|\t)[^\n]+(:?\n[ \t]*[^\s][^\n]*)*)/xm
     subparse:
       context: listClose

data/lib/symbolize.rb

@@ -1,7 +1,4 @@
 #!/usr/bin/env ruby
-#
-#  Created by Larry Baltz
-#  Copyright (c) 2006. All rights reserved.
 
 class Object
   def symbolize; self; end

data/test/MarkdownTest_1.0/Markdown Documentation - Basics.out

@@ -0,0 +1,316 @@
+<h1>Markdown: Basics</h1>
+
+<ul id="ProjectSubmenu">
+    <li><a href="/projects/markdown/" title="Markdown Project Page">Main</a></li>
+    <li><a class="selected" title="Markdown Basics">Basics</a></li>
+    <li><a href="/projects/markdown/syntax" title="Markdown Syntax Documentation">Syntax</a></li>
+    <li><a href="/projects/markdown/license" title="Pricing and License Information">License</a></li>
+    <li><a href="/projects/markdown/dingus" title="Online Markdown Web Form">Dingus</a></li>
+</ul>
+
+<h2>Getting the Gist of Markdown's Formatting Syntax</h2>
+
+<p>This page offers a brief overview of what it's like to use Markdown.
+The <a href="/projects/markdown/syntax" title="Markdown Syntax">syntax page</a> provides complete, detailed documentation for
+every feature, but Markdown should be very easy to pick up simply by
+looking at a few examples of it in action. The examples on this page
+are written in a before/after style, showing example syntax and the
+HTML output produced by Markdown.</p>
+
+<p>It's also helpful to simply try Markdown out; the <a href="/projects/markdown/dingus" title="Markdown Dingus">Dingus</a> is a
+web application that allows you type your own Markdown-formatted text
+and translate it to XHTML.</p>
+
+<p><strong>Note:</strong> This document is itself written using Markdown; you
+can <a href="/projects/markdown/basics.text">see the source for it by adding '.text' to the URL</a>.</p>
+
+
+<h2>Paragraphs, Headers, Blockquotes</h2>
+
+<p>A paragraph is simply one or more consecutive lines of text, separated
+by one or more blank lines. (A blank line is any line that looks like a
+blank line -- a line containing nothing spaces or tabs is considered
+blank.) Normal paragraphs should not be intended with spaces or tabs.</p>
+
+<p>Markdown offers two styles of headers: <em>Setext</em> and <em>atx</em>.
+Setext-style headers for <code>&lt;h1&gt;</code> and <code>&lt;h2&gt;</code> are created by
+"underlining" with equal signs (<code>=</code>) and hyphens (<code>-</code>), respectively.
+To create an atx-style header, you put 1-6 hash marks (<code>#</code>) at the
+beginning of the line -- the number of hashes equals the resulting
+HTML header level.</p>
+
+<p>Blockquotes are indicated using email-style '<code>&gt;</code>' angle brackets.</p>
+
+<p>Markdown:</p>
+
+<pre><code>A First Level Header
+====================
+
+A Second Level Header
+---------------------
+
+Now is the time for all good men to come to
+the aid of their country. This is just a
+regular paragraph.
+
+The quick brown fox jumped over the lazy
+dog's back.
+
+### Header 3
+
+&gt; This is a blockquote.
+&gt; 
+&gt; This is the second paragraph in the blockquote.
+&gt;
+&gt; ## This is an H2 in a blockquote
+</code></pre>
+
+<p>Output:</p>
+
+<pre><code>&lt;h1&gt;A First Level Header&lt;/h1&gt;
+
+&lt;h2&gt;A Second Level Header&lt;/h2&gt;
+
+&lt;p&gt;Now is the time for all good men to come to
+the aid of their country. This is just a
+regular paragraph.&lt;/p&gt;
+
+&lt;p&gt;The quick brown fox jumped over the lazy
+dog's back.&lt;/p&gt;
+
+&lt;h3&gt;Header 3&lt;/h3&gt;
+
+&lt;blockquote&gt;
+    &lt;p&gt;This is a blockquote.&lt;/p&gt;
+    
+    &lt;p&gt;This is the second paragraph in the blockquote.&lt;/p&gt;
+    
+    &lt;h2&gt;This is an H2 in a blockquote&lt;/h2&gt;
+&lt;/blockquote&gt;
+</code></pre>
+
+<h3>Phrase Emphasis</h3>
+
+<p>Markdown uses asterisks and underscores to indicate spans of emphasis.</p>
+
+<p>Markdown:</p>
+
+<pre><code>Some of these words *are emphasized*.
+Some of these words _are emphasized also_.
+
+Use two asterisks for **strong emphasis**.
+Or, if you prefer, __use two underscores instead__.
+</code></pre>
+
+<p>Output:</p>
+
+<pre><code>&lt;p&gt;Some of these words &lt;em&gt;are emphasized&lt;/em&gt;.
+Some of these words &lt;em&gt;are emphasized also&lt;/em&gt;.&lt;/p&gt;
+
+&lt;p&gt;Use two asterisks for &lt;strong&gt;strong emphasis&lt;/strong&gt;.
+Or, if you prefer, &lt;strong&gt;use two underscores instead&lt;/strong&gt;.&lt;/p&gt;
+</code></pre>
+
+<h2>Lists</h2>
+
+<p>Unordered (bulleted) lists use asterisks, pluses, and hyphens (<code>*</code>,
+<code>+</code>, and <code>-</code>) as list markers. These three markers are
+interchangable; this:</p>
+
+<pre><code>*   Candy.
+*   Gum.
+*   Booze.
+</code></pre>
+
+<p>this:</p>
+
+<pre><code>+   Candy.
++   Gum.
++   Booze.
+</code></pre>
+
+<p>and this:</p>
+
+<pre><code>-   Candy.
+-   Gum.
+-   Booze.
+</code></pre>
+
+<p>all produce the same output:</p>
+
+<pre><code>&lt;ul&gt;
+&lt;li&gt;Candy.&lt;/li&gt;
+&lt;li&gt;Gum.&lt;/li&gt;
+&lt;li&gt;Booze.&lt;/li&gt;
+&lt;/ul&gt;
+</code></pre>
+
+<p>Ordered (numbered) lists use regular numbers, followed by periods, as
+list markers:</p>
+
+<pre><code>1.  Red
+2.  Green
+3.  Blue
+</code></pre>
+
+<p>Output:</p>
+
+<pre><code>&lt;ol&gt;
+&lt;li&gt;Red&lt;/li&gt;
+&lt;li&gt;Green&lt;/li&gt;
+&lt;li&gt;Blue&lt;/li&gt;
+&lt;/ol&gt;
+</code></pre>
+
+<p>If you put blank lines between items, you'll get <code>&lt;p&gt;</code> tags for the
+list item text. You can create multi-paragraph list items by indenting
+the paragraphs by 4 spaces or 1 tab:</p>
+
+<pre><code>*   A list item.
+
+    With multiple paragraphs.
+
+*   Another item in the list.
+</code></pre>
+
+<p>Output:</p>
+
+<pre><code>&lt;ul&gt;
+&lt;li&gt;&lt;p&gt;A list item.&lt;/p&gt;
+&lt;p&gt;With multiple paragraphs.&lt;/p&gt;&lt;/li&gt;
+&lt;li&gt;&lt;p&gt;Another item in the list.&lt;/p&gt;&lt;/li&gt;
+&lt;/ul&gt;
+
+</code></pre>
+
+<h3>Links</h3>
+
+<p>Markdown supports two styles for creating links: <em>inline</em> and
+<em>reference</em>. With both styles, you use square brackets to delimit the
+text you want to turn into a link.</p>
+
+<p>Inline-style links use parentheses immediately after the link text.
+For example:</p>
+
+<pre><code>This is an [example link](http://example.com/).
+</code></pre>
+
+<p>Output:</p>
+
+<pre><code>&lt;p&gt;This is an &lt;a href="http://example.com/"&gt;
+example link&lt;/a&gt;.&lt;/p&gt;
+</code></pre>
+
+<p>Optionally, you may include a title attribute in the parentheses:</p>
+
+<pre><code>This is an [example link](http://example.com/ "With a Title").
+</code></pre>
+
+<p>Output:</p>
+
+<pre><code>&lt;p&gt;This is an &lt;a href="http://example.com/" title="With a Title"&gt;
+example link&lt;/a&gt;.&lt;/p&gt;
+</code></pre>
+
+<p>Reference-style links allow you to refer to your links by names, which
+you define elsewhere in your document:</p>
+
+<pre><code>I get 10 times more traffic from [Google][1] than from
+[Yahoo][2] or [MSN][3].
+
+[1]: http://google.com/        "Google"
+[2]: http://search.yahoo.com/  "Yahoo Search"
+[3]: http://search.msn.com/    "MSN Search"
+</code></pre>
+
+<p>Output:</p>
+
+<pre><code>&lt;p&gt;I get 10 times more traffic from &lt;a href="http://google.com/"
+title="Google"&gt;Google&lt;/a&gt; than from &lt;a href="http://search.yahoo.com/"
+title="Yahoo Search"&gt;Yahoo&lt;/a&gt; or &lt;a href="http://search.msn.com/"
+title="MSN Search"&gt;MSN&lt;/a&gt;.&lt;/p&gt;
+</code></pre>
+
+<p>The title attribute is optional. Link names may contain letters,
+numbers and spaces, but are <em>not</em> case sensitive:</p>
+
+<pre><code>I start my morning with a cup of coffee and
+[The New York Times][NY Times].
+
+[ny times]: http://www.nytimes.com/
+</code></pre>
+
+<p>Output:</p>
+
+<pre><code>&lt;p&gt;I start my morning with a cup of coffee and
+&lt;a href="http://www.nytimes.com/"&gt;The New York Times&lt;/a&gt;.&lt;/p&gt;
+</code></pre>
+
+<h3>Images</h3>
+
+<p>Image syntax is very much like link syntax.</p>
+
+<p>Inline (titles are optional):</p>
+
+<pre><code>![alt text](/path/to/img.jpg "Title")
+</code></pre>
+
+<p>Reference-style:</p>
+
+<pre><code>![alt text][id]
+
+[id]: /path/to/img.jpg "Title"
+</code></pre>
+
+<p>Both of the above examples produce the same output:</p>
+
+<pre><code>&lt;img src="/path/to/img.jpg" alt="alt text" title="Title" /&gt;
+</code></pre>
+
+<h3>Code</h3>
+
+<p>In a regular paragraph, you can create code span by wrapping text in
+backtick quotes. Any ampersands (<code>&amp;</code>) and angle brackets (<code>&lt;</code> or
+<code>&gt;</code>) will automatically be translated into HTML entities. This makes
+it easy to use Markdown to write about HTML example code:</p>
+
+<pre><code>I strongly recommend against using any `&lt;blink&gt;` tags.
+
+I wish SmartyPants used named entities like `&amp;mdash;`
+instead of decimal-encoded entites like `&amp;#8212;`.
+</code></pre>
+
+<p>Output:</p>
+
+<pre><code>&lt;p&gt;I strongly recommend against using any
+&lt;code&gt;&amp;lt;blink&amp;gt;&lt;/code&gt; tags.&lt;/p&gt;
+
+&lt;p&gt;I wish SmartyPants used named entities like
+&lt;code&gt;&amp;amp;mdash;&lt;/code&gt; instead of decimal-encoded
+entites like &lt;code&gt;&amp;amp;#8212;&lt;/code&gt;.&lt;/p&gt;
+</code></pre>
+
+<p>To specify an entire block of pre-formatted code, indent every line of
+the block by 4 spaces or 1 tab. Just like with code spans, <code>&amp;</code>, <code>&lt;</code>,
+and <code>&gt;</code> characters will be escaped automatically.</p>
+
+<p>Markdown:</p>
+
+<pre><code>If you want your page to validate under XHTML 1.0 Strict,
+you've got to put paragraph tags in your blockquotes:
+
+    &lt;blockquote&gt;
+        &lt;p&gt;For example.&lt;/p&gt;
+    &lt;/blockquote&gt;
+</code></pre>
+
+<p>Output:</p>
+
+<pre><code>&lt;p&gt;If you want your page to validate under XHTML 1.0 Strict,
+you've got to put paragraph tags in your blockquotes:&lt;/p&gt;
+
+&lt;pre&gt;&lt;code&gt;&amp;lt;blockquote&amp;gt;
+    &amp;lt;p&amp;gt;For example.&amp;lt;/p&amp;gt;
+&amp;lt;/blockquote&amp;gt;
+&lt;/code&gt;&lt;/pre&gt;
+</code></pre>