commonmarker 0.9.2 → 0.10.0

data/ext/commonmarker/cmark/src/scanners.re

@@ -252,7 +252,7 @@ bufsize_t _scan_atx_heading_start(const unsigned char *p)
   const unsigned char *marker = NULL;
   const unsigned char *start = p;
 /*!re2c
-  [#]{1,6} ([ ]+|[\r\n])  { return (bufsize_t)(p - start); }
+  [#]{1,6} ([ \t]+|[\r\n])  { return (bufsize_t)(p - start); }
   .? { return 0; }
 */
 }
@@ -277,9 +277,9 @@ bufsize_t _scan_thematic_break(const unsigned char *p)
   const unsigned char *marker = NULL;
   const unsigned char *start = p;
 /*!re2c
-  ([*][ ]*){3,} [ \t]* [\r\n] { return (bufsize_t)(p - start); }
-  ([_][ ]*){3,} [ \t]* [\r\n] { return (bufsize_t)(p - start); }
-  ([-][ ]*){3,} [ \t]* [\r\n] { return (bufsize_t)(p - start); }
+  ([*][ \t]*){3,} [ \t]* [\r\n] { return (bufsize_t)(p - start); }
+  ([_][ \t]*){3,} [ \t]* [\r\n] { return (bufsize_t)(p - start); }
+  ([-][ \t]*){3,} [ \t]* [\r\n] { return (bufsize_t)(p - start); }
   .? { return 0; }
 */
 }

data/ext/commonmarker/cmark/test/pathological_tests.py

@@ -43,7 +43,7 @@ pathological = {
                   re.compile("(\[ a_){50000}")),
     "hard link/emph case":
                  ("**x [a*b**c*](d)",
-                  re.compile("\\*\\*x <a href=\"d\">a<em>b</em><em>c</em></a>")),
+                  re.compile("\\*\\*x <a href=\"d\">a<em>b\\*\\*c</em></a>")),
     "nested brackets":
                  (("[" * 50000) + "a" + ("]" * 50000),
                   re.compile("\[{50000}a\]{50000}")),

data/ext/commonmarker/cmark/test/spec.txt

@@ -1,8 +1,8 @@
 ---
 title: CommonMark Spec
 author: John MacFarlane
-version: 0.25
-date: '2016-03-24'
+version: 0.26
+date: '2016-07-15'
 license: '[CC-BY-SA 4.0](http://creativecommons.org/licenses/by-sa/4.0/)'
 ...
 
@@ -336,9 +336,14 @@ the Unicode classes `Pc`, `Pd`, `Pe`, `Pf`, `Pi`, `Po`, or `Ps`.
 ## Tabs
 
 Tabs in lines are not expanded to [spaces].  However,
-in contexts where indentation is significant for the
-document's structure, tabs behave as if they were replaced
-by spaces with a tab stop of 4 characters.
+in contexts where whitespace helps to define block structure,
+tabs behave as if they were replaced by spaces with a tab stop
+of 4 characters.
+
+Thus, for example, a tab can be used instead of four spaces
+in an indented code block.  (Note, however, that internal
+tabs are passed through as literal tabs, not expanded to
+spaces.)
 
 ```````````````````````````````` example
 →foo→baz→→bim
@@ -347,7 +352,6 @@ by spaces with a tab stop of 4 characters.
 </code></pre>
 ````````````````````````````````
 
-
 ```````````````````````````````` example
   →foo→baz→→bim
 .
@@ -355,7 +359,6 @@ by spaces with a tab stop of 4 characters.
 </code></pre>
 ````````````````````````````````
 
-
 ```````````````````````````````` example
     a→a
     ὐ→a
@@ -365,6 +368,9 @@ by spaces with a tab stop of 4 characters.
 </code></pre>
 ````````````````````````````````
 
+In the following example, a continuation paragraph of a list
+item is indented with a tab; this has exactly the same effect
+as indentation with four spaces would:
 
 ```````````````````````````````` example
   - foo
@@ -393,6 +399,15 @@ by spaces with a tab stop of 4 characters.
 </ul>
 ````````````````````````````````
 
+Normally the `>` that begins a block quote may be followed
+optionally by a space, which is not considered part of the
+content.  In the following case `>` is followed by a tab,
+which is treated as if it were expanded into spaces.
+Since one of theses spaces is considered part of the
+delimiter, `foo` is considered to be indented six spaces
+inside the block quote context, so we get an indented
+code block starting with two spaces.
+
 ```````````````````````````````` example
 >→→foo
 .
@@ -441,6 +456,17 @@ bar
 </ul>
 ````````````````````````````````
 
+```````````````````````````````` example
+#→Foo
+.
+<h1>Foo</h1>
+````````````````````````````````
+
+```````````````````````````````` example
+*→*→*→
+.
+<hr />
+````````````````````````````````
 
 
 ## Insecure characters
@@ -779,15 +805,6 @@ headings:
 ````````````````````````````````
 
 
-A tab will not work:
-
-```````````````````````````````` example
-#→foo
-.
-<p>#→foo</p>
-````````````````````````````````
-
-
 This is not a heading, because the first `#` is escaped:
 
 ```````````````````````````````` example
@@ -1968,7 +1985,7 @@ by their start and end conditions.  The block begins with a line that
 meets a [start condition](@) (after up to three spaces
 optional indentation).  It ends with the first subsequent line that
 meets a matching [end condition](@), or the last line of
-the document, if no line is encountered that meets the
+the document or other [container block](@), if no line is encountered that meets the
 [end condition].  If the first line meets both the [start condition]
 and the [end condition], the block will contain just that line.
 
@@ -2302,6 +2319,7 @@ import Text.HTML.TagSoup
 main :: IO ()
 main = print $ parseTags tags
 </code></pre>
+okay
 .
 <pre language="haskell"><code>
 import Text.HTML.TagSoup
@@ -2309,6 +2327,7 @@ import Text.HTML.TagSoup
 main :: IO ()
 main = print $ parseTags tags
 </code></pre>
+<p>okay</p>
 ````````````````````````````````
 
 
@@ -2320,12 +2339,14 @@ A script tag (type 1):
 
 document.getElementById("demo").innerHTML = "Hello JavaScript!";
 </script>
+okay
 .
 <script type="text/javascript">
 // JavaScript example
 
 document.getElementById("demo").innerHTML = "Hello JavaScript!";
 </script>
+<p>okay</p>
 ````````````````````````````````
 
 
@@ -2338,6 +2359,7 @@ h1 {color:red;}
 
 p {color:blue;}
 </style>
+okay
 .
 <style
   type="text/css">
@@ -2345,6 +2367,7 @@ h1 {color:red;}
 
 p {color:blue;}
 </style>
+<p>okay</p>
 ````````````````````````````````
 
 
@@ -2433,11 +2456,13 @@ A comment (type 2):
 
 bar
    baz -->
+okay
 .
 <!-- Foo
 
 bar
    baz -->
+<p>okay</p>
 ````````````````````````````````
 
 
@@ -2450,12 +2475,14 @@ A processing instruction (type 3):
   echo '>';
 
 ?>
+okay
 .
 <?php
 
   echo '>';
 
 ?>
+<p>okay</p>
 ````````````````````````````````
 
 
@@ -2483,6 +2510,7 @@ function matchwo(a,b)
   }
 }
 ]]>
+okay
 .
 <![CDATA[
 function matchwo(a,b)
@@ -2496,6 +2524,7 @@ function matchwo(a,b)
   }
 }
 ]]>
+<p>okay</p>
 ````````````````````````````````
 
 
@@ -3240,8 +3269,8 @@ Four spaces gives us a code block:
 ````````````````````````````````
 
 
-The Laziness clause allows us to omit the `>` before a
-paragraph continuation line:
+The Laziness clause allows us to omit the `>` before
+[paragraph continuation text]:
 
 ```````````````````````````````` example
 > # Foo
@@ -3347,8 +3376,8 @@ foo
 ````````````````````````````````
 
 
-Note that in the following case, we have a paragraph
-continuation line:
+Note that in the following case, we have a [lazy
+continuation line]:
 
 ```````````````````````````````` example
 > foo
@@ -3370,7 +3399,7 @@ To see why, note that in
 
 the `- bar` is indented too far to start a list, and can't
 be an indented code block because indented code blocks cannot
-interrupt paragraphs, so it is a [paragraph continuation line].
+interrupt paragraphs, so it is [paragraph continuation text].
 
 A block quote can be empty:
 
@@ -3599,7 +3628,7 @@ The following rules define [list items]:
 1.  **Basic case.**  If a sequence of lines *Ls* constitute a sequence of
     blocks *Bs* starting with a [non-whitespace character] and not separated
     from each other by more than one blank line, and *M* is a list
-    marker of width *W* followed by 0 < *N* < 5 spaces, then the result
+    marker of width *W* followed by 1 ≤ *N* ≤ 4 spaces, then the result
     of prepending *M* and the following spaces to the first line of
     *Ls*, and indenting subsequent lines of *Ls* by *W + N* spaces, is a
     list item with *Bs* as its contents.  The type of the list item
@@ -3607,6 +3636,12 @@ The following rules define [list items]:
     If the list item is ordered, then it is also assigned a start
     number, based on the ordered list marker.
 
+    Exceptions: When the list item interrupts a paragraph---that
+    is, when it starts on a line that would otherwise count as
+    [paragraph continuation text]---then (a) the lines *Ls* must
+    not begin with a blank line, and (b) if the list item is
+    ordered, the start number must be 1.
+
 For example, let *Ls* be the lines
 
 ```````````````````````````````` example
@@ -3781,66 +3816,20 @@ any following content, so these are not list items:
 ````````````````````````````````
 
 
-A list item may not contain blocks that are separated by more than
-one blank line.  Thus, two blank lines will end a list, unless the
-two blanks are contained in a [fenced code block].
+A list item may contain blocks that are separated by more than
+one blank line.
 
 ```````````````````````````````` example
-- foo
-
-  bar
-
 - foo
 
 
   bar
-
-- ```
-  foo
-
-
-  bar
-  ```
-
-- baz
-
-  + ```
-    foo
-
-
-    bar
-    ```
 .
 <ul>
 <li>
 <p>foo</p>
 <p>bar</p>
 </li>
-<li>
-<p>foo</p>
-</li>
-</ul>
-<p>bar</p>
-<ul>
-<li>
-<pre><code>foo
-
-
-bar
-</code></pre>
-</li>
-<li>
-<p>baz</p>
-<ul>
-<li>
-<pre><code>foo
-
-
-bar
-</code></pre>
-</li>
-</ul>
-</li>
 </ul>
 ````````````````````````````````
 
@@ -3873,15 +3862,14 @@ A list item may contain any kind of block:
 
 
 A list item that contains an indented code block will preserve
-empty lines within the code block verbatim, unless there are two
-or more empty lines in a row (since as described above, two
-blank lines end the list):
+empty lines within the code block verbatim.
 
 ```````````````````````````````` example
 - Foo
 
       bar
 
+
       baz
 .
 <ul>
@@ -3889,33 +3877,13 @@ blank lines end the list):
 <p>Foo</p>
 <pre><code>bar
 
-baz
-</code></pre>
-</li>
-</ul>
-````````````````````````````````
-
-
-```````````````````````````````` example
-- Foo
 
-      bar
-
-
-      baz
-.
-<ul>
-<li>
-<p>Foo</p>
-<pre><code>bar
+baz
 </code></pre>
 </li>
 </ul>
-<pre><code>  baz
-</code></pre>
 ````````````````````````````````
 
-
 Note that ordered list start numbers must be nine digits or less:
 
 ```````````````````````````````` example
@@ -4242,6 +4210,20 @@ A list may start or end with an empty list item:
 </ul>
 ````````````````````````````````
 
+However, an empty list item cannot interrupt a paragraph:
+
+```````````````````````````````` example
+foo
+*
+
+foo
+1.
+.
+<p>foo
+*</p>
+<p>foo
+1.</p>
+````````````````````````````````
 
 
 4.  **Indentation.**  If a sequence of lines *Ls* constitutes a list item
@@ -4439,13 +4421,18 @@ So, in this case we need two spaces indent:
 - foo
   - bar
     - baz
+      - boo
 .
 <ul>
 <li>foo
 <ul>
 <li>bar
 <ul>
-<li>baz</li>
+<li>baz
+<ul>
+<li>boo</li>
+</ul>
+</li>
 </ul>
 </li>
 </ul>
@@ -4460,11 +4447,13 @@ One is not enough:
 - foo
  - bar
   - baz
+   - boo
 .
 <ul>
 <li>foo</li>
 <li>bar</li>
 <li>baz</li>
+<li>boo</li>
 </ul>
 ````````````````````````````````
 
@@ -4817,28 +4806,20 @@ Foo
 </ul>
 ````````````````````````````````
 
-
 `Markdown.pl` does not allow this, through fear of triggering a list
 via a numeral in a hard-wrapped line:
 
-```````````````````````````````` example
+```````````````````````````````` markdown
 The number of windows in my house is
 14.  The number of doors is 6.
-.
-<p>The number of windows in my house is</p>
-<ol start="14">
-<li>The number of doors is 6.</li>
-</ol>
 ````````````````````````````````
+Oddly, though, `Markdown.pl` *does* allow a blockquote to
+interrupt a paragraph, even though the same considerations might
+apply.
 
-
-Oddly, `Markdown.pl` *does* allow a blockquote to interrupt a paragraph,
-even though the same considerations might apply.  We think that the two
-cases should be treated the same.  Here are two reasons for allowing
-lists to interrupt paragraphs:
-
-First, it is natural and not uncommon for people to start lists without
-blank lines:
+In CommonMark, we do allow lists to interrupt paragraphs, for
+two reasons.  First, it is natural and not uncommon for people
+to start lists without blank lines:
 
     I need to buy
     - new shoes
@@ -4872,20 +4853,40 @@ then
 
 by itself should be a paragraph followed by a nested sublist.
 
-Our adherence to the [principle of uniformity]
-thus inclines us to think that there are two coherent packages:
+Since it is well established Markdown practice to allow lists to
+interrupt paragraphs inside list items, the [principle of
+uniformity] requires us to allow this outside list items as
+well.  ([reStructuredText](http://docutils.sourceforge.net/rst.html)
+takes a different approach, requiring blank lines before lists
+even inside other list items.)
 
-1.  Require blank lines before *all* lists and blockquotes,
-    including lists that occur as sublists inside other list items.
+In order to solve of unwanted lists in paragraphs with
+hard-wrapped numerals, we allow only lists starting with `1` to
+interrupt paragraphs.  Thus,
 
-2.  Require blank lines in none of these places.
+```````````````````````````````` example
+The number of windows in my house is
+14.  The number of doors is 6.
+.
+<p>The number of windows in my house is
+14.  The number of doors is 6.</p>
+````````````````````````````````
 
-[reStructuredText](http://docutils.sourceforge.net/rst.html) takes
-the first approach, for which there is much to be said.  But the second
-seems more consistent with established practice with Markdown.
+We may still get an unintended result in cases like
 
-There can be blank lines between items, but two blank lines end
-a list:
+```````````````````````````````` example
+The number of windows in my house is
+1.  The number of doors is 6.
+.
+<p>The number of windows in my house is</p>
+<ol>
+<li>The number of doors is 6.</li>
+</ol>
+````````````````````````````````
+
+but this rule should prevent most spurious list captures.
+
+There can be any number of blank lines between items:
 
 ```````````````````````````````` example
 - foo
@@ -4902,36 +4903,12 @@ a list:
 <li>
 <p>bar</p>
 </li>
-</ul>
-<ul>
-<li>baz</li>
-</ul>
-````````````````````````````````
-
-
-As illustrated above in the section on [list items],
-two blank lines between blocks *within* a list item will also end a
-list:
-
-```````````````````````````````` example
-- foo
-
-
-  bar
-- baz
-.
-<ul>
-<li>foo</li>
-</ul>
-<p>bar</p>
-<ul>
-<li>baz</li>
+<li>
+<p>baz</p>
+</li>
 </ul>
 ````````````````````````````````
 
-
-Indeed, two blank lines will end *all* containing lists:
-
 ```````````````````````````````` example
 - foo
   - bar
@@ -4945,26 +4922,28 @@ Indeed, two blank lines will end *all* containing lists:
 <ul>
 <li>bar
 <ul>
-<li>baz</li>
+<li>
+<p>baz</p>
+<p>bim</p>
+</li>
 </ul>
 </li>
 </ul>
 </li>
 </ul>
-<pre><code>  bim
-</code></pre>
 ````````````````````````````````
 
 
-Thus, two blank lines can be used to separate consecutive lists of
-the same type, or to separate a list from an indented code block
-that would otherwise be parsed as a subparagraph of the final list
-item:
+To separate consecutive lists of the same type, or to separate a
+list from an indented code block that would otherwise be parsed
+as a subparagraph of the final list item, you can insert a blank HTML
+comment:
 
 ```````````````````````````````` example
 - foo
 - bar
 
+<!-- -->
 
 - baz
 - bim
@@ -4973,6 +4952,7 @@ item:
 <li>foo</li>
 <li>bar</li>
 </ul>
+<!-- -->
 <ul>
 <li>baz</li>
 <li>bim</li>
@@ -4987,6 +4967,7 @@ item:
 
 -   foo
 
+<!-- -->
 
     code
 .
@@ -4999,6 +4980,7 @@ item:
 <p>foo</p>
 </li>
 </ul>
+<!-- -->
 <pre><code>code
 </code></pre>
 ````````````````````````````````
@@ -5945,18 +5927,22 @@ The following rules define emphasis and strong emphasis:
 
 9.  Emphasis begins with a delimiter that [can open emphasis] and ends
     with a delimiter that [can close emphasis], and that uses the same
-    character (`_` or `*`) as the opening delimiter.  There must
-    be a nonempty sequence of inlines between the open delimiter
-    and the closing delimiter; these form the contents of the emphasis
-    inline.
+    character (`_` or `*`) as the opening delimiter.  The
+    opening and closing delimiters must belong to separate
+    [delimiter runs].  If one of the delimiters can both
+    open and close emphasis, then the sum of the lengths of the
+    delimiter runs containing the opening and closing delimiters
+    must not be a multiple of 3.
 
 10. Strong emphasis begins with a delimiter that
     [can open strong emphasis] and ends with a delimiter that
     [can close strong emphasis], and that uses the same character
-    (`_` or `*`) as the opening delimiter.
-    There must be a nonempty sequence of inlines between the open
-    delimiter and the closing delimiter; these form the contents of
-    the strong emphasis inline.
+    (`_` or `*`) as the opening delimiter.  The
+    opening and closing delimiters must belong to separate
+    [delimiter runs].  If one of the delimiters can both open
+    and close strong emphasis, then the sum of the lengths of
+    the delimiter runs containing the opening and closing
+    delimiters must not be a multiple of 3.
 
 11. A literal `*` character cannot occur at the beginning or end of
     `*`-delimited emphasis or `**`-delimited strong emphasis, unless it
@@ -5980,9 +5966,7 @@ the following principles resolve ambiguity:
     so that the second begins before the first ends and ends after
     the first ends, the first takes precedence. Thus, for example,
     `*foo _bar* baz_` is parsed as `<em>foo _bar</em> baz_` rather
-    than `*foo <em>bar* baz</em>`.  For the same reason,
-    `**foo*bar**` is parsed as `<em><em>foo</em>bar</em>*`
-    rather than `<strong>foo*bar</strong>`.
+    than `*foo <em>bar* baz</em>`.
 
 16. When there are two potential emphasis or strong emphasis spans
     with the same closing delimiter, the shorter one (the one that
@@ -6155,10 +6139,8 @@ A newline also counts as whitespace:
 *foo bar
 *
 .
-<p>*foo bar</p>
-<ul>
-<li></li>
-</ul>
+<p>*foo bar
+*</p>
 ````````````````````````````````
 
 
@@ -6562,18 +6544,30 @@ __foo_ bar_
 <p><em>foo <strong>bar</strong> baz</em></p>
 ````````````````````````````````
 
-
-But note:
-
 ```````````````````````````````` example
 *foo**bar**baz*
 .
-<p><em>foo</em><em>bar</em><em>baz</em></p>
+<p><em>foo<strong>bar</strong>baz</em></p>
 ````````````````````````````````
 
+Note that in the preceding case, the interpretation
+
+``` markdown
+<p><em>foo</em><em>bar<em></em>baz</em></p>
+```
+
+
+is precluded by the condition that a delimiter that
+can both open and close (like the `*` after `foo`
+cannot form emphasis if the sum of the lengths of
+the delimiter runs containing the opening and
+closing delimiters is a multiple of 3.
+
+The same condition ensures that the following
+cases are all strong emphasis nested inside
+emphasis, even when the interior spaces are
+omitted:
 
-The difference is that in the preceding case, the internal delimiters
-[can close emphasis], while in the cases with spaces, they cannot.
 
 ```````````````````````````````` example
 ***foo** bar*
@@ -6589,17 +6583,18 @@ The difference is that in the preceding case, the internal delimiters
 ````````````````````````````````
 
 
-Note, however, that in the following case we get no strong
-emphasis, because the opening delimiter is closed by the first
-`*` before `bar`:
-
 ```````````````````````````````` example
 *foo**bar***
 .
-<p><em>foo</em><em>bar</em>**</p>
+<p><em>foo<strong>bar</strong></em></p>
 ````````````````````````````````
 
 
+```````````````````````````````` example
+*foo**bar***
+.
+<p><em>foo<strong>bar</strong></em></p>
+````````````````````````````````
 
 Indefinite levels of nesting are possible:
 
@@ -6693,18 +6688,13 @@ ____foo__ bar__
 ````````````````````````````````
 
 
-But note:
-
 ```````````````````````````````` example
 **foo*bar*baz**
 .
-<p><em><em>foo</em>bar</em>baz**</p>
+<p><strong>foo<em>bar</em>baz</strong></p>
 ````````````````````````````````
 
 
-The difference is that in the preceding case, the internal delimiters
-[can close emphasis], while in the cases with spaces, they cannot.
-
 ```````````````````````````````` example
 ***foo* bar**
 .
@@ -7019,13 +7009,6 @@ Rule 15:
 ````````````````````````````````
 
 
-```````````````````````````````` example
-**foo*bar**
-.
-<p><em><em>foo</em>bar</em>*</p>
-````````````````````````````````
-
-
 ```````````````````````````````` example
 *foo __bar *baz bim__ bam*
 .
@@ -7956,7 +7939,7 @@ consists of a [link label] that [matches] a
 [link reference definition] elsewhere in the
 document and is not followed by `[]` or a link label.
 The contents of the first link label are parsed as inlines,
-which are used as the link's text.  the link's URI and title
+which are used as the link's text.  The link's URI and title
 are provided by the matching link reference definition.
 Thus, `[foo]` is equivalent to `[foo][]`.
 
@@ -8931,7 +8914,7 @@ foo
 
 A regular line break (not in a code span or HTML tag) that is not
 preceded by two or more spaces or a backslash is parsed as a
-softbreak.  (A softbreak may be rendered in HTML either as a
+[softbreak](@).  (A softbreak may be rendered in HTML either as a
 [line ending] or as a space. The result will be the same in
 browsers. In the examples here, a [line ending] will be used.)