markascend 0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: da8df76e1fec2a44dfd1dcaa2b8b27d7f12a86fe
+  data.tar.gz: bfd019319d0b249bb1bfca43a3aea1b8dd9943f9
+SHA512:
+  metadata.gz: d53e8a4039c10b9f74eebd2be669a48b9b328859b53f2397e7b356bfece745ec5f15183959a50856713fa0cc15aad04a4569e596d94de3500b0d49b3b300e156
+  data.tar.gz: f6199c6c3052ad1fd5db6d7001c193b0be5550e6b8c0592a6dee94e48d07a142557b05bbb2a6749eb0b7ece9fb07fa898a1bafbb656faa143920efe657caffc6

data/Gemfile

@@ -0,0 +1,9 @@
+source 'https://rubygems.org'
+
+gem 'rake'
+gem 'slim', '~> 2.0.0.pre.6'
+gem 'nokogiri'
+gem 'pry'
+gem 'pygments.rb'
+gem 'compass'
+gem 'filemagic'

data/copying

@@ -0,0 +1,18 @@
+Copyright (C) 2013 by Zete Lui (BSD)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN

data/doc/api.ma

@@ -0,0 +1,65 @@
+\hi(ruby)
+h2 Compile
+
+|
+  Markascend.compile src, options
+
+h2 Options
+
+- `:autolink`, default value is `%w[http https ftp mailto]`
+- `:inline_img`, compile image into inlined base64, default = `false`
+- `:macros`, specify the names of enabled macros. Other macros will be treated as plain text. The default value is `Markascend::DEFAULT_MACROS`.
+- `:line_units`, specify the inline parsers to be used (be careful with the order!). The default value is `Markascend::DEFAULT_LINE_UNITS`.
+- `:sandbox`, a hybrid option to tweak the syntax to be generally safe for user inputs. `false` by default. When set to `true`, footnotes are disabled, header anchors are ignored, and enabled macros are set to `Markascend::SANDBOX_MACROS`. The sandbox macro list can be overriden by the `:macros` option.
+- `:toc`, whether generate table of contents. `false` by default. Header anchors can be customized or generated in the form of `"-#{N}"`. Note that there's no "permalink" generator for headers, but you can implement one with simple javascript.
+
+h2 Customizing macros
+
+More macro processors can be added by
+
+|
+  class Markascend::Macro
+    def parse_fancy_macro
+      ... compose result string with: env, content, inline
+    end
+  end
+
+Macro names are limited to names like ruby methods.
+
+|
+  Markascend.compile src, macros: %w[fancy_macro del]
+
+h2 Customizing line-unit parsers
+
+More line-unit parsers can be added by
+
+|
+  class Markascend::LineUnit
+    def parse_at
+      ... compose result string with: env, line, block, linenum
+    end
+  end
+
+The list of inline parsers can be changed, or reordered
+
+|
+  Markascend.compile src, line_units: Markascend::DEFAULT_LINE_UNITS + %w[at]
+
+h2 Notes on `\slim`
+
+You need to install slim \hi(bash)(`gem ins slim`) and require it before using the \hi(ma)`\slim` macro:
+\hi(ruby)
+|
+  require 'slim'
+
+It is disabled in sandbox mode.
+
+h2 Notes on `\dot`
+
+You need to install [graphviz](graphviz.org) before using the \hi(ma)`\dot` macro.
+
+It is disabled in sandbox mode.
+
+h2 Notes on output format
+
+The output is valid HTML5, but not XHTML.

data/doc/index.ma

@@ -0,0 +1,23 @@
+h2 Introduction
+
+Markascend is an extensible markdown-like, macro-powered html document syntax and processor.
+Can be used as blog-generator, post-formatter or for literate programing.
+
+h2 Install
+
+Requires Ruby 2.0.0+
+
+|sh
+  gem ins markascend
+
+h2 Cheat Sheet
+
+h2 Document
+
+- [Syntax in Details](syntax)
+- [API in Details](api)
+- [Editor Support (Textmate/Sublime)](https://github.com/luikore/Markascend.tmbundle)
+
+h2 License
+
+[BSD](https://github.com/luikore/markascend/tree/master/copying)

data/doc/syntax.ma

@@ -0,0 +1,264 @@
+\options
+  title: Syntax
+
+h2 Design principles
+
+- A simpler *Markdown* should be better.
+- Since a markup language usually need to process several different syntaces in one article, language composition becomes important in syntax design.
+- TeX macro syntax is nearly perfect for extending the language, but not very visual intuitive when there are too much backslashes and bracers.
+- Layout-based grammar (indentation sensitive) shows a cleaner way for language composition and cleaning up macros.
+
+\hi(ma)
+
+h2 Inline elements
+*italic*
+|
+  *italic*
+
+**bold**
+|
+  **bold**
+
+Italic and bold elements can interpolate
+|
+  ***bold and italic*bold**
+
+`code` suject to the same rule as in markdown, no escape chars
+|
+  `code`
+  ``to include "`", use more backticks as delimiter``
+  `` ` space before and after to disambig backtick number ``
+
+$math$, inside which `\\` and `\$` are treated as atomic elements, no escape chars
+|
+  $math$
+  $\\\$$ <- the content is parsed to latex processer as '\\\$'
+  $\\$$ <- NOTE this is a math elem with a pending dollar
+
+Link:
+|
+  url in lexical parens: [a](example.com/use/\\/for/backslash/and/\)/for/right/paren)
+  url in recursive braces: [a]{example.com/allow/{nested{braces}}/and/(parens)/in/url}
+  no need to escape backslashes in braces: [a]{file:\\\C:\windows}
+  link to anchor: [a](#anchor)
+
+Footnotes:
+|
+  [.](this is shown in footnote and replaced by a number)
+  [.*](this is shown in footnote and replaced by the acronym: *)
+
+Using already defined footnote ("one more dot to use"):
+|
+  [:1]
+  [:link by acronym]
+
+Except math and code, inline elements can be nested.
+
+By default, text starting with `http://`, `https://` or `mailto://` will be auto-linked, you can change supported protocols or disable all with compile options.
+
+NOTE: Not very different from markdown here, just less, math and footnotes added. Images are too complex for a simple syntax, it is made into a macro.
+
+h2 Headers
+|
+  h1 title
+  h2#anchor-name title
+
+NOTE: Anchors are disabled in sandbox mode.
+h2 Lists
+
+`-` is the bullet for unordered list
+|
+  - an entry
+  - another entry
+
+`+` is the bullet for ordered list
+|
+  + entry 1
+  + entry 2
+
+h2 Quote
+
+Just one `>` is enough for a quote containing many lines (NOTE: use the similar rule as code block and allow quoter name?)
+|
+  > You
+    can
+    input
+    new line
+    on twitter
+    now
+
+h2 Escape characters
+|
+  \h3
+  \\
+  \$
+  \*
+  \[
+  \%
+  \@
+  \#
+  ...
+
+All HTML4 entities are supported
+|
+  &amp;
+  &#2382;
+  &#x003F;
+
+h2 Built-in macros
+
+The following macros explains themselves:
+|
+  \del{deleted}
+  \underline{underline}
+  \sub{subscript}
+  \sup{supscript}
+  \img{hello/a.png alt="an image" href="a.b.c"}
+
+Unlike in *Mardown*, all html tags in *Markascend* are escaped. If you write
+|ma
+  <div></div>
+
+You get HTML like this
+|html
+  &lt;div&gt;&lt;/div&gt;
+
+To put HTML in, you can use the `\html` macro
+|
+  \html{<hr>}
+
+*Slim* is a template engine which makes html much cleaner:
+|
+  \slim{a href="/" Home}
+
+Note that with `\slim`, you can embed a bunch of other template engines, or write ruby logic, or do all kinds of evil things:
+|
+  \slim
+    - 3.times do
+      markdown:
+        # h1 hello world
+    sass:
+      h1
+        color: red
+    coffee:
+      alert "hello world"
+
+To build a table from a CSV
+|
+  \csv
+    name,type
+    "Ahri",Mage
+    "Miss Fortune",DPS
+
+A table from CSV, without header
+|
+  \headless_csv
+    "Ahri",Mage
+    "Miss Fortune",DPS
+
+For block-styled math
+|
+  \latex
+    \begin{align} \\
+    & \rm{MathJax}\ \LaTeX \ \rm{Example} \\ \\
+    \end{align}
+
+`\options` is a special macro. The content *YAML* is not processed by *Markascend*, but retrievable with API. This is designed for blogger generators: you can put metadata inside the document.
+|
+  \options
+    tags: [markascend readme]
+
+To change default code syntax highliter (See [below](#syn-hi-macros) for details]:
+|
+  \hi(rb)
+  \hi(none)
+
+h2 Notes on macros
+
+Macro name is case sensitive, starts with a word-character but not digit, and contains no symbols or punctuations. The rule expressed in (Onigmo) regexp is `(?!\d)\w+`. For example, `你2` is valid name, but `2你` is invalid.
+
+Macros can be of inline form (`\macro{content}`) or block form (`\macro` and the indented block in following lines is the content of the macro).
+
+Inline macros can use some different delimiters for convinience, allowed ones are: `\macro(content)` and `\macro{content}`.
+
+In the form `\macro(content)`, the content is parsed by lexical rule. The parser handles and only handles 2 escape chars: `\\` and `\)`, very much like single quoted string in Ruby.
+Example: `\macro(\ and \\ are both single-backslash. nest: (\))`
+
+In the form `\macro{content}`, you don't escape chars in the content, and can put in recursively nested braces. When you need to put an unmatched `{` or `}` in the content, use the previous form.
+Example: `\macro{\\ is two backslashes. nest: {}}`
+
+Design NOTE: There are actual very few symbols left for us... for example `[]`, `$$`, `|` may confuse with link or math or code elements. Symbols like `"`, `'`, `()`, `<>` are not good either, because they are common when composing an article.
+
+User NOTE:
+- The parsing rules for `(content)` and `{content}` are the same as in links.
+- Only 1 block-style macro name is allowed inside a line.
+- The basic parsing unit is a line and a block, every parsing starts with this unit --- so don't hand-add space indentations to paragraphs, let css do it.
+
+h2 Charting macros
+
+You need to install [graphviz](http://graphviz.org/) first.
+
+|
+  \dot
+    digraph G{
+      main -> parse;
+    }
+
+h2 Popular-company macros
+
+This is the transient part but provides some convienience. The list will change if some of them dies before *Markascend*.
+
+To generate a twitter or weibo link
+|
+  \twitter(@night_song)
+  \twitter(#HotS)
+  \weibo(@yavaeye)
+
+To generate a wiki link (though wiki is an organization, not a company...)
+|
+  \wiki{ruby(programing_language)}
+
+Embed gist
+|
+  \gist(12321)
+
+Embed video (currently should recognize youtube, vimeo and niconico links, the size is a problem)
+|
+  \video(500x400 http://www.youtube.com/watch?v=TGPvtlnwH6E)
+  \video(500x400 http://www.youtube.com/watch?v=TGPvtlnwH6E)
+  \video(500x400 http://www.youtube.com/watch?v=TGPvtlnwH6E)
+
+h2#syn-hi-macros Syntax hilite macros and code blocks
+
+Inline code can also have syntax hiliter
+
+|
+  \hi(rb)`str.size` in ruby is equivalent to \hi(objc)`str.length` in objc
+  |
+    this part is also hilited in objc
+  \hi(none)`\hi(none)` removes hilite state in document
+  |
+    this part also has no syntax hilite
+
+To specify syntax in block form (this won't change default syntax):
+|
+  |rb
+    puts 'hello world'
+
+h2 Combining macro blocks, lists and quotes
+
+Lists and quotes are parsed as recursive block elements.
+If the inside contains macros, need to indent more for the content of the macro:
+|
+  > Though the \music macro..
+      1232. 1232. 1232.
+    is not defined.
+
+Another nesting example:
+|
+  - an entry
+    contains an ordered list
+    + entry \latex
+        \mathbb{one}
+    + entry 2
+  - another entry

data/lib/markascend.rb

@@ -0,0 +1,172 @@
+require "strscan"
+require "cgi"
+require "csv"
+require "yaml"
+require "base64"
+require "open3"
+require "pygments"
+require "open-uri"
+require "filemagic"
+
+module Markascend
+  VERSION = '0.1'
+
+  DEFAULT_MACROS = Hash.[] %w[
+    del underline sub sup
+    img html slim
+    csv headless_csv
+    latex
+    options hi
+    dot
+
+    twitter weibo
+    wiki
+    gist
+    video
+  ].map{|k| [k, "parse_#{k}"]}
+
+  SANDBOX_MACROS = DEFAULT_MACROS.dup.delete_if do |k, v|
+    %w[html slim options dot].include? k
+  end
+
+  # NOTE on the order:
+  # - link/bold/italic can contain char
+  #   but link need not interpolate with bold or italic, seems too rare cased
+  # - bold/italic can interpolate with each other
+  # - escapes are processed in the char parser, link/bold/italic can use escape chars
+  DEFAULT_LINE_UNITS = %w[
+    inline_code
+    math
+    auto_link
+    macro
+    link
+    bold_italic
+    char
+  ].map{|k| "parse_#{k}"}
+
+  class << Markascend
+    def compile src, opts={}
+      src = src.gsub "\t", '  '
+      env = Env.new opts
+      res = Parser.new(env, src).parse
+
+      if env.toc and !env.toc.empty?
+        res = (generate_toc(env.toc) << res)
+      end
+
+      if env.footnotes and !env.footnotes.empty?
+        res << generate_footnotes(env.footnotes)
+      end
+
+      res
+    end
+
+    attr_accessor :inline_parsers, :macros
+
+    # escape html
+    def escape_html s
+      CGI.escape_html s
+    end
+
+    # escape string so that the result can be placed inside double-quoted value of a tag property
+    def escape_attr s
+      # http://www.w3.org/TR/html5/syntax.html#attributes-0
+      s ? (s.gsub /"/, '&quot;') : ''
+    end
+
+    # escape string so that the result can be placed inside a `<pre>` tag
+    def escape_pre s
+      s.gsub(/(<)|(>)|&/){$1 ? '&lt;' : $2 ? '&gt;' : '&amp;'}
+    end
+
+    # syntax hilite s with lang
+    def hilite s, lang, inline=false
+      if !lang or lang =~ /\A(ma(rkascend)?)?\z/i or !(::Pygments::Lexer.find lang)
+        # TODO ma lexer
+        s = inline ? (escape_html s) : (escape_pre s)
+      else
+        s = Pygments.highlight s, lexer: lang, options: {nowrap: true}
+      end
+
+      # TODO config class
+      if inline
+        %Q|<code class="highlight">#{s}</code>|
+      else
+        %Q|<pre><code class="highlight">#{s}</code></pre>|
+      end
+    end
+
+    # detect mime type of the buffer
+    def mime buffer
+      fm = FileMagic.new FileMagic::MAGIC_MIME_TYPE
+      res = fm.buffer buffer
+      fm.close
+      res
+    end
+
+    # strip tags from s
+    def strip_tags s
+      # deal with html tags only
+      s.gsub(/
+        \<\s*script\b
+          (?:
+            (["']).*?\1|[^\>] # properties
+          )*
+        \>
+        .*?
+        \<\s*\/\s*script\s*\>
+      /x, '').gsub(/
+        \<\s*(?:\/\s*)?
+          \w+\b               # tag name, no need to care xml namespace
+          (?:
+            (["']).*?\1|[^\>] # properties
+          )*
+        \>
+      /x, '')
+    end
+
+    private
+
+    def generate_toc toc
+      res = '<div id="toc"><ol>'
+      levels = toc.values.map(&:first).uniq.sort
+      prev_level = 0
+      toc.each do |id, (x, header_content)| # x as in "hx"
+        level = levels.index(x)
+        if level >= prev_level
+          (level - prev_level).times do
+            res << "<ol>"
+          end
+        elsif
+          (prev_level - level).times do
+            res << "</ol>"
+          end
+        end
+        prev_level = level
+        title = strip_tags header_content
+        res << %Q{<li><a href="\##{id}">#{title}</a></li>}
+      end
+      prev_level.times do
+        res << "</ol>"
+      end
+      res << "</ol></div>"
+    end
+
+    def generate_footnotes footnotes
+      res = '<div id="footnotes"><ol>'
+      i = 0
+      footnotes.each do |abbrev, detail|
+        res << %Q|<li id="footnote-#{i}">#{escape_html detail}</li>|
+        i += 1
+      end
+      res << "</ol></div>"
+    end
+  end
+end
+
+require_relative "markascend/env"
+require_relative "markascend/parser"
+require_relative "markascend/line_unit"
+require_relative "markascend/macro"
+require_relative "markascend/builtin_macros"
+require_relative "markascend/popular_company_macros"