ebnf 2.0.0 → 2.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 06a44ff62e466914a2d6f1d80d4eacae30745c5c7823799af847db98cd27c813
-  data.tar.gz: fa433cd2719ec088cadfd987d69effa5490a3219166bd93406b1e73537956056
+  metadata.gz: dc55292610eb978d5751361069f3b993d35db3a597442e2027cf0fd2ff886ba5
+  data.tar.gz: cc74cd0257a36fa3591f54becfdb51dfffbf44662598f2d67c6a36bf4e969e61
 SHA512:
-  metadata.gz: 335fb057c452beb803c64fdfcbe87bc908e26323904a9641892987f819beb87edea7c3ca2f2bfa1aa911fcbf4b642c41b5cfdf5ba77568a8455283482fcaeb7e
-  data.tar.gz: f4dd5ab8c0b86c2cf2ca2b1edc4f5d93ce3c86a6b5bb527f43def66c33374090873be59ecadb60874bea62d5f80bfa76c230dc7468caf5e8af3557125d887b85
+  metadata.gz: bf7c7df32e027a0739b4830651dcff1f4b5186ff2177e1f57b4e952db33619660c4025a29ffc8dba5c7d0f5f5b95f2cbd432a379bc2ffb02d22f7ec6913a48e2
+  data.tar.gz: 909a8ff438172431054a33fb067198e32d98d5b88fa630978831af4f1dd728f0197512ee1f341667eb3039662db618122d7da5d0cc4cc55838625f685f7d7a9b

data/README.md

@@ -9,10 +9,17 @@
 ## Description
 This is a [Ruby][] implementation of an [EBNF][] and [BNF][] parser and parser generator.
 
+### [PEG][]/[Packrat][] Parser
+In the primary mode, it supports a Parsing Expression Grammar ([PEG][]) parser generator. This performs more minmal transformations on the parsed grammar to extract sub-productions, which allows each component of a rule to generate its own parsing event.
+
+The resulting {EBNF::PEG::Rule} objects then parse each associated rule according to the operator semantics and use a [Packrat][] memoizer to reduce extra work when backtracking.
+
+These rules are driven using the {EBNF::PEG::Parser} module which calls invokes the starting rule and ensures that all input is consumed.
+
 ### LL(1) Parser
-In one mode, it parses [EBNF][] grammars to [BNF][], generates [First/Follow][] and Branch tables for [LL(1)][] grammars, which can be used with the stream [Tokenizer][] and [LL(1) Parser][].
+In another mode, it parses [EBNF][] grammars to [BNF][], generates [First/Follow][] and Branch tables for [LL(1)][] grammars, which can be used with the stream [Tokenizer][] and [LL(1) Parser][].
 
-As LL(1) grammars operate using `alt` and `seq` primitives, allowing for a match on alternative productions or a sequence of productions, generating a parser requires turning the EBNF rules into BNF:
+As LL(1) grammars operate using `alt` and `seq` primitives, allowing for a match on alternative productions or a sequence of productions, generating a parser requires turning the [EBNF][] rules into [BNF][]:
 
 * Transform `a ::= b?` into `a ::= _empty | b`
 * Transform `a ::= b+` into `a ::= b b*`
@@ -25,58 +32,77 @@ As LL(1) grammars operate using `alt` and `seq` primitives, allowing for a match
 
 Of note in this implementation is that the tokenizer and parser are streaming, so that they can process inputs of arbitrary size.
 
-See {EBNF::LL1} and {EBNF::LL1::Parser} for further information.
+The _exception operator_ (`A - B`) is only supported on terminals.
 
-### [PEG][]/[Packrat][] Parser
-An additional Parsing Expression Grammar ([PEG][]) parser generator is also supported. This performs more minmal transformations on the parsed grammar to extract sub-productions, which allows each component of a rule to generate its own parsing event.
+See {EBNF::LL1} and {EBNF::LL1::Parser} for further information.
 
 ## Usage
 ### Parsing an EBNF Grammar
 
     require 'ebnf'
 
-    ebnf = EBNF.parse(File.open('./etc/ebnf.ebnf'))
+    grammar = EBNF.parse(File.open('./etc/ebnf.ebnf'))
 
-Output rules and terminals as S-Expressions, Turtle, HTML or BNF
+Output rules and terminals as [S-Expressions][S-Expression], [Turtle][], HTML or [BNF][]
 
-    puts ebnf.to_sxp
-    puts ebnf.to_ttl
-    puts ebnf.to_html
-    puts ebnf.to_s
+    puts grammar.to_sxp
+    puts grammar.to_ttl
+    puts grammar.to_html
+    puts grammar.to_s
 
-Transform EBNF to PEG (generates sub-rules for embedded expressions) and the RULES table as Ruby for parsing grammars:
+Transform [EBNF][] to [PEG][] (generates sub-rules for embedded expressions) and the RULES table as Ruby for parsing grammars:
 
-    ebnf.make_peg
-    ebnf.to_ruby
+    grammar.make_peg
+    grammar.to_ruby
 
-Transform EBNF to BNF (generates sub-rules using `alt` or `seq` from `plus`, `star` or `opt`)
+Transform [EBNF][] to [BNF][] (generates sub-rules using `alt` or `seq` from `plus`, `star` or `opt`)
 
-    ebnf.make_bnf
+    grammar.make_bnf
 
 Generate [First/Follow][] rules for BNF grammars (using "ebnf" as the starting production):
 
-    ebnf.first_follow(:ebnf)
+    grammar.first_follow(:ebnf)
 
 Generate Terminal, [First/Follow][], Cleanup and Branch tables as Ruby for parsing grammars:
 
-    ebnf.build_tables
-    ebnf.to_ruby
+    grammar.build_tables
+    grammar.to_ruby
 
 Generate formatted grammar using HTML (requires [Haml][Haml] gem):
 
-    ebnf.to_html
+    grammar.to_html
+
+### Parsing an ISO/IEC 14977 Grammar
+
+The EBNF gem can also parse [ISO/EIC 14977] Grammars (ISOEBNF) to [S-Expressions][S-Expression].
+
+    grammar = EBNF.parse(File.open('./etc/iso-ebnf.isoebnf', format: :isoebnf))
+
+### Parsing an ABNF Grammar
+
+The EBNF gem can also parse [ABNF] Grammars to [S-Expressions][S-Expression].
+
+    grammar = EBNF.parse(File.open('./etc/abnf.abnf', format: :abnf))
 
-### Parser debugging
+### Parser Debugging
 
 Inevitably while implementing a parser for some specific grammar, a developer will need greater insight into the operation of the parser. While this can involve sorting through a tremendous amount of data, the parser can be provided a [Logger][] instance which will output messages at varying levels of detail to document the state of the parser at any given point. Most useful is likely the `INFO` level of debugging, but even more detail is revealed using the `DEBUG` level. `WARN` and `ERROR` statements will typically also be provided as part of an exception if parsing fails, but can be shown in the context of other parsing state with appropriate indentation as part of the logger.
 
-### Parser errors
+### Writing Grammars
+
+The {EBNF::Writer} class can be used to write parsed grammars out, either as formatted text, or HTML. Because grammars are written from the Abstract Syntax Tree, represented as [S-Expressions][S-Expression], this provides a means of transforming between grammar formats (e.g., W3C [EBNF][] to [ABNF][]), although with some potential loss in semantic fidelity (case-insensitive string matching vs. case-sensitive matching).
+
+The formatted HTML results are designed to be appropriate for including in specifications.
+
+### Parser Errors
 On a parsing failure, and exception is raised with information that may be useful in determining the source of the error.
 
 ## EBNF Grammar
 The [EBNF][] variant used here is based on [W3C](https://w3.org/) [EBNF][] (see {file:etc/ebnf.ebnf EBNF grammar}) as defined in the
 [XML 1.0 recommendation](https://www.w3.org/TR/REC-xml/), with minor extensions:
 
+The character set for EBNF is UTF-8.
+
 The general form of a rule is:
 
     symbol ::= expression
@@ -85,7 +111,9 @@ which can also be proceeded by an optional number enclosed in square brackets to
 
     [1] symbol ::= expression
 
-Symbols are written with an initial capital letter if they are the start symbol of a regular language (terminals), otherwise with an initial lowercase letter (non-terminals). Literal strings are quoted.
+(Note, this can introduce an ambiguity if the previous rule ends in a range or enum and the current rule has no identifier. In this case, enclosing `expression` within parentheses, or adding intervening comments can resolve the ambiguity.)
+
+Symbols are written in CAPITAL CASE if they are the start symbol of a regular language (terminals), otherwise with they are treated as non-terminal rules. Literal strings are quoted.
 
 Within the expression on the right-hand side of a rule, the following expressions are used to match strings of one or more characters:
 
@@ -93,13 +121,13 @@ Within the expression on the right-hand side of a rule, the following expression
   <tr><td><code>#xN</code></td>
     <td>where <code>N</code> is a hexadecimal integer, the expression matches the character whose number (code point) in ISO/IEC 10646 is <code>N</code>. The number of leading zeros in the <code>#xN</code> form is insignificant.</td></tr>
   <tr><td><code>[a-zA-Z], [#xN-#xN]</code>
-    <td>matches any Char with a value in the range(s) indicated (inclusive).</td></tr>
+    <td>matches any Char or HEX with a value in the range(s) indicated (inclusive).</td></tr>
   <tr><td><code>[abc], [#xN#xN#xN]</code></td>
-    <td>matches any Char with a value among the characters enumerated. Enumerations and ranges can be mixed in one set of brackets.</td></tr>
+    <td>matches any UTF-8 R\_CHAR or HEX with a value among the characters enumerated. The last component may be '-'. Enumerations and ranges may be mixed in one set of brackets.</td></tr>
   <tr><td><code>[^a-z], [^#xN-#xN]</code></td>
-    <td>matches any Char with a value outside the range indicated.</td></tr>
+    <td>matches any UTF-8 Char or HEX a value outside the range indicated.</td></tr>
   <tr><td><code>[^abc], [^#xN#xN#xN]</code></td>
-    <td>matches any Char with a value not among the characters given. Enumerations and ranges of forbidden values can be mixed in one set of brackets.</td></tr>
+    <td>matches any UTF-8 R\_CHAR or HEX with a value not among the characters given. The last component may be '-'. Enumerations and ranges of excluded values may be mixed in one set of brackets.</td></tr>
   <tr><td><code>"string"</code></td>
     <td>matches a literal string matching that given inside the double quotes.</td></tr>
   <tr><td><code>'string'</code></td>
@@ -113,7 +141,7 @@ Within the expression on the right-hand side of a rule, the following expression
   <tr><td><code>A | B</code></td>
     <td>matches <code>A</code> or <code>B</code>.</td></tr>
   <tr><td><code>A - B</code></td>
-    <td>matches any string that matches <code>A</code> but does not match <code>B</code>.</td></tr>
+    <td>matches any string that matches <code>A</code> but does not match <code>B</code>. (Only supported on Terminals in LL(1) BNF).</td></tr>
   <tr><td><code>A+</code></td>
     <td>matches one or more occurrences of <code>A</code>. Concatenation has higher precedence than alternation; thus <code>A+ | B+</code> is identical to <code>(A+) | (B+)</code>.</td></tr>
   <tr><td><code>A*</code></td>
@@ -130,10 +158,10 @@ Within the expression on the right-hand side of a rule, the following expression
 * `@pass` defines the expression used to detect whitespace, which is removed in processing.
 * No support for `wfc` (well-formedness constraint) or `vc` (validity constraint).
 
-Parsing this grammar yields an S-Expression version: {file:etc/ebnf.sxp} (or [LL(1)][] version {file:etc/ebnf.ll1.sxp} or [PEG][] version {file:etc/ebnf.peg.sxp}).
+Parsing this grammar yields an [S-Expression][] version: {file:etc/ebnf.sxp} (or [LL(1)][] version {file:etc/ebnf.ll1.sxp} or [PEG][] version {file:etc/ebnf.peg.sxp}).
 
 ### Parser S-Expressions
-Intermediate representations of the grammar may be serialized to Lisp-like S-Expressions. For example, the rule
+Intermediate representations of the grammar may be serialized to Lisp-like [S-Expressions][S-Expression]. For example, the rule
 
     [1] ebnf        ::= (declaration | rule)*
 
@@ -155,13 +183,23 @@ Different components of an EBNF rule expression are transformed into their own o
   <tr><td><code>A?</code></td><td><code>(opt A)</code></td></tr>
   <tr><td><code>A B</code></td><td><code>(seq A B)</code></td></tr>
   <tr><td><code>A | B</code></td><td><code>(alt A B)</code></td></tr>
-  <tr><td><code>A - B</code></td><td><code>(diff A B)</code></td></tr>
+  <tr><td><code>A - B</code></td>
+    <td><code>(diff A B) for terminals.<br/>
+      <code>(seq (not B) A) for non-terminals (PEG parsing only)</code></code></td></tr>
   <tr><td><code>A+</code></td><td><code>(plus A)</code></td></tr>
   <tr><td><code>A*</code></td><td><code>(star A)</code></td></tr>
-  <tr><td><code>@pass " "*</code></td><td><code>(pass (star " "))</code></td></tr>
+  <tr><td><code>@pass " "*</code></td><td><code>(pass _pass (star " "))</code></td></tr>
   <tr><td><code>@terminals</code></td><td></td></tr>
 </table>
 
+Other rule operators are not directly supported in [EBNF][], but are included to support other notations (e.g., [ABNF][] and [ISO/IEC 14977][]):
+
+<table>
+  <tr><td><code>%i"StRiNg"</code></td><td><code>(istr "StRiNg")</code></td><td>Case-insensitive string matching</td></tr>
+  <tr><td><code>'' - A</code></td><td><code>(not A)</code></td><td>Negative look-ahead, used for non-terminal uses of `B - A`.</td></tr>
+  <tr><td><code>n*mA</code></td><td><code>(rept n m A)</code></td><td>Explicit repetition.</td></tr>
+</table>
+
 Additionally, rules defined with an UPPERCASE symbol are treated as terminals.
 
 For an [LL(1)][] parser generator, the {EBNF::BNF.make_bnf} method can be used to transform the EBNF rule into a BNF rule.
@@ -179,12 +217,16 @@ For a [PEG][] parser generator, there is a simpler transformation that reduces r
     (rule _ebnf_1 "1.1" (alt declaration rule))
 
 ## Example parsers
-For a [PEG][] parser for a simple grammar implementing a calculator see [Calc example](http://dryruby.github.io/ebnf/examples/calc/doc/calc.html
+For a [PEG][] parser for a simple grammar implementing a calculator see [Calc example](https://dryruby.github.io/ebnf/examples/calc/doc/calc.html)
 
-For an example parser built using this gem that parses the [EBNF][] grammar, see [EBNF PEG Parser example](http://dryruby.github.io/ebnf/examples/ebnf-peg-parser/doc/parser.html). This example creates a parser for the [EBNF][] grammar which generates the same Abstract Syntax Tree as the built-in parser in the gem.
+For an example parser built using this gem that parses the [EBNF][] grammar, see [EBNF PEG Parser example](https://dryruby.github.io/ebnf/examples/ebnf-peg-parser/doc/parser.html). This example creates a parser for the [EBNF][] grammar which generates the same Abstract Syntax Tree as the built-in parser in the gem.
 
 There is also an
-[EBNF LL(1) Parser example](http://dryruby.github.io/ebnf/examples/ebnf-peg-parser/doc/parser.html).
+[EBNF LL(1) Parser example](https://dryruby.github.io/ebnf/examples/ebnf-peg-parser/doc/parser.html).
+
+The [ISO EBNF Parser](https://dryruby.github.io/ebnf/examples/isoebnf/doc/parser.html) example parses [ISO/IEC 14977][] into [S-Expressions][S-Expression], which can be used to parse compatible grammars using this parser (either [PEG][] or [LL(1)][]).
+
+The [ABNF Parser](https://dryruby.github.io/ebnf/examples/abnf/doc/parser.html) example parses [ABNF][] into [S-Expressions][S-Expression], which can be used to parse compatible grammars using this [PEG][] parser.
 
 ##  Acknowledgements
 Much of this work, particularly the generic parser, is inspired by work originally done by
@@ -229,16 +271,19 @@ A copy of the [Turtle EBNF][] and derived parser files are included in the repos
 [YARD]:         https://yardoc.org/
 [YARD-GS]:      https://rubydoc.info/docs/yard/file/docs/GettingStarted.md
 [PDD]:          https://lists.w3.org/Archives/Public/public-rdf-ruby/2010May/0013.html
+[ABNF]:         https://www.rfc-editor.org/rfc/rfc5234
 [BNF]:          https://en.wikipedia.org/wiki/Backus–Naur_form
 [EBNF]:         https://www.w3.org/TR/REC-xml/#sec-notation
 [EBNF doc]:     https://rubydoc.info/github/dryruby/ebnf
 [First/Follow]: https://en.wikipedia.org/wiki/LL_parser#Constructing_an_LL.281.29_parsing_table
+[ISO/IEC 14977]:https://www.iso.org/standard/26153.html
 [LL(1)]:        https://www.csd.uwo.ca/~moreno//CS447/Lectures/Syntax.html/node14.html
 [LL(1) Parser]: https://en.wikipedia.org/wiki/LL_parser
 [Logger]:       https://ruby-doc.org/stdlib-2.4.0/libdoc/logger/rdoc/Logger.html
+[S-expression]: https://en.wikipedia.org/wiki/S-expression
 [Tokenizer]:    https://en.wikipedia.org/wiki/Lexical_analysis#Tokenizer
+[Turtle]:       https://www.w3.org/TR/2012/WD-turtle-20120710/
 [Turtle EBNF]:  https://dvcs.w3.org/hg/rdf/file/default/rdf-turtle/turtle.bnf
 [Packrat]:      https://pdos.csail.mit.edu/~baford/packrat/thesis/
 [PEG]:          https://en.wikipedia.org/wiki/Parsing_expression_grammar
-[Treetop]:      https://rubygems.org/gems/treetop
 [Haml]:         https://rubygems.org/gems/haml

data/VERSION

@@ -1 +1 @@
-2.0.0
+2.1.0

data/bin/ebnf

@@ -15,6 +15,7 @@ options = {
   output_format: :sxp,
   prefix: "ttl",
   namespace: "http://www.w3.org/ns/formats/Turtle#",
+  level: 4
 }
 
 input, out = nil, STDOUT
@@ -23,15 +24,17 @@ OPT_ARGS = [
   ["--debug",           GetoptLong::NO_ARGUMENT,      "Turn on debugging output"],
   ["--bnf",             GetoptLong::NO_ARGUMENT,      "Transform EBNF to BNF"],
   ["--evaluate","-e",   GetoptLong::REQUIRED_ARGUMENT,"Evaluate argument as an EBNF document"],
+  ["--format", "-f",    GetoptLong::REQUIRED_ARGUMENT,"Specify output format one of abnf, abnfh, ebnf, html, isoebnf, isoebnfh, ttl, sxp, or rb"],
+  ["--input-format",    GetoptLong::REQUIRED_ARGUMENT,"Specify input format one of abnf, ebnf isoebnf, native, or sxp"],
   ["--ll1",             GetoptLong::REQUIRED_ARGUMENT,"Generate First/Follow rules, argument is start symbol"],
-  ["--format", "-f",    GetoptLong::REQUIRED_ARGUMENT,"Specify output format one of ebnf, html, ttl, sxp, or rb"],
-  ["--input-format",    GetoptLong::REQUIRED_ARGUMENT,"Specify input format one of ebnf or sxp"],
   ["--mod-name",        GetoptLong::REQUIRED_ARGUMENT,"Module name used when creating ruby tables"],
+  ["--namespace", "-n", GetoptLong::REQUIRED_ARGUMENT,"Namespace to use when generating Turtle"],
   ["--output", "-o",    GetoptLong::REQUIRED_ARGUMENT,"Output to the specified file path"],
   ["--peg",             GetoptLong::NO_ARGUMENT,      "Transform EBNF to PEG"],
   ["--prefix", "-p",    GetoptLong::REQUIRED_ARGUMENT,"Prefix to use when generating Turtle"],
   ["--progress", "-v",  GetoptLong::NO_ARGUMENT,      "Detail on execution"],
-  ["--namespace", "-n", GetoptLong::REQUIRED_ARGUMENT,"Namespace to use when generating Turtle"],
+  ["--renumber",        GetoptLong::NO_ARGUMENT,      "Renumber parsed reules"],
+  ["--validate",        GetoptLong::NO_ARGUMENT,      "Validate grammar"],
   ["--help", "-?",      GetoptLong::NO_ARGUMENT,      "This message"]
 ]
 def usage
@@ -54,27 +57,34 @@ opts = GetoptLong.new(*OPT_ARGS.map {|o| o[0..-2]})
 
 opts.each do |opt, arg|
   case opt
-  when '--debug'        then options[:debug] = true
+  when '--debug'        then options[:level] = 0
   when '--bnf'          then options[:bnf] = true
   when '--evaluate'     then input = arg
-  when '--input-format' then options[:format] = arg.to_sym
-  when '--format'       then options[:output_format] = arg.to_sym
+  when '--input-format'
+    unless %w(abnf ebnf isoebnf native sxp).include?(arg)
+      STDERR.puts("unrecognized input format #{arg}")
+      usage
+    end
+    options[:format] = arg.to_sym
+  when '--format'
+    unless %w(abnf abnfh ebnf html isoebnf isoebnfh rb sxp).include?(arg)
+      STDERR.puts("unrecognized output format #{arg}")
+      usage
+    end
+    options[:output_format] = arg.to_sym
   when '--ll1'          then (options[:ll1] ||= []) << arg.to_sym
   when '--mod-name'     then options[:mod_name] = arg
   when '--output'       then out = File.open(arg, "w")
   when '--peg'          then options[:peg] = true
   when '--prefix'       then options[:prefix] = arg
+  when '--renumber'     then options[:renumber] = true
   when '--namespace'    then options[:namespace] = arg
-  when '--progress'     then options[:progress] = true
+  when '--progress'     then options[:level] = 1 unless options[:level] == 0
+  when '--validate'     then options[:validate] = true
   when '--help'         then usage
   end
 end
 
-if options[:output_format] == :rb && !(options[:ll1] || options[:peg])
-  STDERR.puts "outputing in .rb format requires --ll or --peg"
-  exit(1)
-end
-
 input = File.open(ARGV[0]) if ARGV[0]
 
 ebnf = EBNF.parse(input || STDIN, **options)
@@ -85,13 +95,19 @@ if options[:ll1]
   ebnf.build_tables
 end
 
+ebnf.renumber! if options[:renumber]
+
 res = case options[:output_format]
-when :ebnf    then ebnf.to_s
-when :html    then ebnf.to_html
-when :sxp     then ebnf.to_sxp
-when :ttl     then ebnf.to_ttl(options[:prefix], options[:namespace])
-when :rb      then ebnf.to_ruby(out, grammarFile: ARGV[0], **options)
-else          ebnf.ast.inspect
+when :abnf      then ebnf.to_s(format: :abnf)
+when :abnfh     then ebnf.to_html(format: :abnf)
+when :ebnf      then ebnf.to_s
+when :html      then ebnf.to_html
+when :isoebnf   then ebnf.to_s(format: :isoebnf)
+when :isoebnfh  then ebnf.to_html(format: :isoebnf)
+when :sxp       then ebnf.to_sxp
+when :ttl       then ebnf.to_ttl(options[:prefix], options[:namespace])
+when :rb        then ebnf.to_ruby(out, grammarFile: ARGV[0], **options)
+else            ebnf.ast.inspect
 end
 
 out.puts res

data/etc/abnf-core.ebnf

@@ -0,0 +1,52 @@
+# Core terminals available in uses of ABNF
+ALPHA        ::=  [#x41-#x5A#x61-#x7A]   # A-Z | a-z
+
+BIT          ::=  '0' | '1'
+
+CHAR         ::=  [#x01-#x7F]
+                       # any 7-bit US-ASCII character,
+                       #  excluding NUL
+CR           ::=  #x0D
+                       # carriage return
+
+CRLF         ::=  CR? LF
+                       # Internet standard newline
+
+CTL          ::=  [#x00-#x1F] | #x7F
+                       # controls
+
+DIGIT        ::=  [#x30-#x39]
+                       # 0-9
+
+DQUOTE       ::=  #x22
+                       # " (Double Quote)
+
+HEXDIG       ::=  DIGIT | [A-F] # [0-9A-F]
+
+HTAB         ::=  #x09
+                       # horizontal tab
+
+LF           ::=  #x0A
+                       # linefeed
+
+LWSP         ::=  (WSP | CRLF WSP)*
+                       # Use of this linear-white-space rule
+                       #  permits lines containing only white
+                       #  space that are no longer legal in
+                       #  mail headers and have caused
+                       #  interoperability problems in other
+                       #  contexts.
+                       # Do not use when defining mail
+                       #  headers and use with caution in
+                       #  other contexts.
+
+OCTET        ::=  [#x00-#xFF]
+                       # 8 bits of data
+
+SP           ::=  #x20
+
+VCHAR        ::=  [#x21-#x7E]
+                       # visible (printing) characters
+
+WSP          ::=  SP | HTAB
+                       # white space

data/etc/abnf.abnf

@@ -0,0 +1,121 @@
+rulelist       =  1*( rule / (*c-wsp c-nl) )
+
+rule           =  rulename defined-as elements c-nl
+                      ; continues if next line starts
+                      ;  with white space
+
+rulename       =  ALPHA *(ALPHA / DIGIT / "-")
+
+defined-as     =  *c-wsp ("=" / "=/") *c-wsp
+                      ; basic rules definition and
+                      ;  incremental alternatives
+
+elements       =  alternation *c-wsp
+
+c-wsp          =  WSP / (c-nl WSP)
+
+c-nl           =  comment / CRLF
+                      ; comment or newline
+
+comment        =  ";" *(WSP / VCHAR) CRLF
+
+alternation    =  concatenation
+                  *(*c-wsp "/" *c-wsp concatenation)
+
+concatenation  =  repetition *(1*c-wsp repetition)
+
+repetition     =  [repeat] element
+
+repeat         =  (*DIGIT "*" *DIGIT) / 1*DIGIT
+
+element        =  rulename / group / option /
+                  char-val / num-val / prose-val
+
+group          =  "(" *c-wsp alternation *c-wsp ")"
+
+option         =  "[" *c-wsp alternation *c-wsp "]"
+
+char-val       =  case-insensitive-string /
+                  case-sensitive-string
+
+case-insensitive-string =
+                  [ "%i" ] quoted-string
+
+case-sensitive-string =
+                  "%s" quoted-string
+
+quoted-string  =  DQUOTE *(%x20-21 / %x23-7E) DQUOTE
+                       ; quoted string of SP and VCHAR
+                       ;  without DQUOTE
+
+num-val        =  "%" (bin-val / dec-val / hex-val)
+
+bin-val        =  "b" 1*BIT
+                  [ 1*("." 1*BIT) / ("-" 1*BIT) ]
+                       ; series of concatenated bit values
+                       ;  or single ONEOF range
+
+dec-val        =  "d" 1*DIGIT
+                  [ 1*("." 1*DIGIT) / ("-" 1*DIGIT) ]
+
+hex-val        =  "x" 1*HEXDIG
+                  [ 1*("." 1*HEXDIG) / ("-" 1*HEXDIG) ]
+
+prose-val      =  "<" *(%x20-3D / %x3F-7E) ">"
+                       ; bracketed string of SP and VCHAR
+                       ;  without angles
+                       ; prose description, to be used as
+                       ;  last resort
+
+ALPHA          =  %x41-5A / %x61-7A   ; A-Z / a-z
+
+BIT            =  "0" / "1"
+
+CHAR           =  %x01-7F
+                         ; any 7-bit US-ASCII character,
+                         ;  excluding NUL
+CR             =  %x0D
+                             ; carriage return
+
+CRLF           =  [CR] LF
+                       ; Internet standard newline
+                       ; Extended to allow only newline
+
+CTL            =  %x00-1F / %x7F
+                       ; controls
+
+DIGIT          =  %x30-39
+                       ; 0-9
+
+DQUOTE         =  %x22
+                       ; " (Double Quote)
+
+HEXDIG         =  DIGIT / "A" / "B" / "C" / "D" / "E" / "F"
+
+HTAB           =  %x09
+                       ; horizontal tab
+
+LF             =  %x0A
+                       ; linefeed
+
+LWSP           =  *(WSP / CRLF WSP)
+                       ; Use of this linear-white-space rule
+                       ;  permits lines containing only white
+                       ;  space that are no longer legal in
+                       ;  mail headers and have caused
+                       ;  interoperability problems in other
+                       ;  contexts.
+                       ; Do not use when defining mail
+                       ;  headers and use with caution in
+                       ;  other contexts.
+
+OCTET          =  %x00-FF
+                       ; 8 bits of data
+
+SP             =  %x20
+
+VCHAR          =  %x21-7E
+                       ; visible (printing) characters
+
+WSP            =  SP / HTAB
+                       ; white space