ebnf 1.1.3 → 2.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (56) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +221 -198
  3. data/UNLICENSE +1 -1
  4. data/VERSION +1 -1
  5. data/bin/ebnf +40 -21
  6. data/etc/abnf-core.ebnf +52 -0
  7. data/etc/abnf.abnf +121 -0
  8. data/etc/abnf.ebnf +124 -0
  9. data/etc/abnf.sxp +45 -0
  10. data/etc/doap.ttl +23 -15
  11. data/etc/ebnf.ebnf +21 -33
  12. data/etc/ebnf.html +171 -160
  13. data/etc/{ebnf.rb → ebnf.ll1.rb} +30 -107
  14. data/etc/ebnf.ll1.sxp +182 -183
  15. data/etc/ebnf.peg.rb +90 -0
  16. data/etc/ebnf.peg.sxp +84 -0
  17. data/etc/ebnf.sxp +40 -41
  18. data/etc/iso-ebnf.ebnf +140 -0
  19. data/etc/iso-ebnf.isoebnf +138 -0
  20. data/etc/iso-ebnf.sxp +65 -0
  21. data/etc/sparql.ebnf +4 -4
  22. data/etc/sparql.html +1603 -1751
  23. data/etc/sparql.ll1.sxp +7372 -7372
  24. data/etc/sparql.peg.rb +532 -0
  25. data/etc/sparql.peg.sxp +597 -0
  26. data/etc/sparql.sxp +363 -362
  27. data/etc/turtle.ebnf +3 -3
  28. data/etc/turtle.html +465 -517
  29. data/etc/{turtle.rb → turtle.ll1.rb} +3 -4
  30. data/etc/turtle.ll1.sxp +425 -425
  31. data/etc/turtle.peg.rb +182 -0
  32. data/etc/turtle.peg.sxp +199 -0
  33. data/etc/turtle.sxp +103 -101
  34. data/lib/ebnf.rb +7 -2
  35. data/lib/ebnf/abnf.rb +301 -0
  36. data/lib/ebnf/abnf/core.rb +23 -0
  37. data/lib/ebnf/abnf/meta.rb +111 -0
  38. data/lib/ebnf/base.rb +113 -69
  39. data/lib/ebnf/bnf.rb +1 -26
  40. data/lib/ebnf/ebnf/meta.rb +90 -0
  41. data/lib/ebnf/isoebnf.rb +229 -0
  42. data/lib/ebnf/isoebnf/meta.rb +75 -0
  43. data/lib/ebnf/ll1.rb +138 -6
  44. data/lib/ebnf/ll1/lexer.rb +37 -32
  45. data/lib/ebnf/ll1/parser.rb +113 -73
  46. data/lib/ebnf/ll1/scanner.rb +83 -51
  47. data/lib/ebnf/native.rb +320 -0
  48. data/lib/ebnf/parser.rb +285 -302
  49. data/lib/ebnf/peg.rb +39 -0
  50. data/lib/ebnf/peg/parser.rb +561 -0
  51. data/lib/ebnf/peg/rule.rb +250 -0
  52. data/lib/ebnf/rule.rb +443 -148
  53. data/lib/ebnf/terminals.rb +21 -0
  54. data/lib/ebnf/writer.rb +565 -83
  55. metadata +107 -29
  56. data/etc/sparql.rb +0 -45773
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 658c0da6f1ba2b472e94badcedabc0dabfab7b3cbce02e58fe6bfb286416e247
4
- data.tar.gz: 9cc79c8f8c0ce5d4e24eb8e1dfbac9d52b1bfe26951c97fd435a547fdb80cf84
3
+ metadata.gz: f58817e075b6100c10ce82eb2da850a71aa9ff76c8d1c359d819720c89477564
4
+ data.tar.gz: 67fb9fc3ae589ae8ab49fef81a5703958635e7b3f8169ccc8df2d17e57c62074
5
5
  SHA512:
6
- metadata.gz: b0cce6804a7d9afa85fe0559148110e2e3443242ff44fac31f77c5939a22158b5bd1fe7bdb5474a98e2221b474c0010bf1e93612e9bd01320212406557892469
7
- data.tar.gz: 41f3a9b7b7bbaa650afa269f54a56724b029e6b14ac31a62632b8d09ce62578624221b15836f4fe52a549408b79d7bd8a20b841d9d6141bb18546efe211b0c59
6
+ metadata.gz: 13874b6745062e294d15176bba4c482ccdc32974a2733429cb1bc23eb87f608ae6e2deeba4e972bdf236629783c3443d7c8461c90e0db4fc6ef6b39d2564582c
7
+ data.tar.gz: 172c8fa0efc03357f845cabff8d06fb11260cd271cb5a084350f8f432beef8d560cc285cef6ab87f4526d5331d8c4c433ae2b9536e108d2a4e0a7bd522a3292b
data/README.md CHANGED
@@ -2,15 +2,24 @@
2
2
 
3
3
  [EBNF][] parser and generic parser generator.
4
4
 
5
- [![Gem Version](https://badge.fury.io/rb/ebnf.png)](http://badge.fury.io/rb/ebnf)
6
- [![Build Status](https://secure.travis-ci.org/dryruby/ebnf.png?branch=master)](http://travis-ci.org/dryruby/ebnf)
5
+ [![Gem Version](https://badge.fury.io/rb/ebnf.png)](https://badge.fury.io/rb/ebnf)
6
+ [![Build Status](https://secure.travis-ci.org/dryruby/ebnf.png?branch=master)](https://travis-ci.org/dryruby/ebnf)
7
7
  [![Coverage Status](https://coveralls.io/repos/dryruby/ebnf/badge.svg)](https://coveralls.io/r/dryruby/ebnf)
8
- [![Dependency Status](https://gemnasium.com/dryruby/ebnf.png)](https://gemnasium.com/dryruby/ebnf)
9
8
 
10
9
  ## Description
11
- This is a [Ruby][] implementation of an [EBNF][] and [BNF][] parser and parser generator. It parses [EBNF][] grammars to [BNF][], generates [First/Follow][] and Branch tables for [LL(1)][] grammars, which can be used with the stream [Tokenizer][] and [LL(1) Parser][].
10
+ This is a [Ruby][] implementation of an [EBNF][] and [BNF][] parser and parser generator.
12
11
 
13
- As LL(1) grammars operate using `alt` and `seq` primitives, allowing for a match on alternative productions or a sequence of productions, generating a parser requires turning the EBNF rules into BNF:
12
+ ### [PEG][]/[Packrat][] Parser
13
+ In the primary mode, it supports a Parsing Expression Grammar ([PEG][]) parser generator. This performs more minmal transformations on the parsed grammar to extract sub-productions, which allows each component of a rule to generate its own parsing event.
14
+
15
+ The resulting {EBNF::PEG::Rule} objects then parse each associated rule according to the operator semantics and use a [Packrat][] memoizer to reduce extra work when backtracking.
16
+
17
+ These rules are driven using the {EBNF::PEG::Parser} module which calls invokes the starting rule and ensures that all input is consumed.
18
+
19
+ ### LL(1) Parser
20
+ In another mode, it parses [EBNF][] grammars to [BNF][], generates [First/Follow][] and Branch tables for [LL(1)][] grammars, which can be used with the stream [Tokenizer][] and [LL(1) Parser][].
21
+
22
+ As LL(1) grammars operate using `alt` and `seq` primitives, allowing for a match on alternative productions or a sequence of productions, generating a parser requires turning the [EBNF][] rules into [BNF][]:
14
23
 
15
24
  * Transform `a ::= b?` into `a ::= _empty | b`
16
25
  * Transform `a ::= b+` into `a ::= b b*`
@@ -23,213 +32,221 @@ As LL(1) grammars operate using `alt` and `seq` primitives, allowing for a match
23
32
 
24
33
  Of note in this implementation is that the tokenizer and parser are streaming, so that they can process inputs of arbitrary size.
25
34
 
35
+ The _exception operator_ (`A - B`) is only supported on terminals.
36
+
37
+ See {EBNF::LL1} and {EBNF::LL1::Parser} for further information.
38
+
26
39
  ## Usage
27
- ### Parsing an LL(1) Grammar
40
+ ### Parsing an EBNF Grammar
28
41
 
29
42
  require 'ebnf'
30
43
 
31
- ebnf = EBNF.parse(File.open('./etc/ebnf.ebnf'))
44
+ grammar = EBNF.parse(File.open('./etc/ebnf.ebnf'))
32
45
 
33
- Output rules and terminals as S-Expressions, Turtle, HTML or BNF
46
+ Output rules and terminals as [S-Expressions][S-Expression], [Turtle][], HTML or [BNF][]
34
47
 
35
- puts ebnf.to_sxp
36
- puts ebnf.to_ttl
37
- puts ebnf.to_html
38
- puts ebnf.to_s
48
+ puts grammar.to_sxp
49
+ puts grammar.to_ttl
50
+ puts grammar.to_html
51
+ puts grammar.to_s
39
52
 
40
- Transform EBNF to BNF (generates sub-productions using `alt` or `seq` from `plus`, `star` or `opt`)
53
+ Transform [EBNF][] to [PEG][] (generates sub-rules for embedded expressions) and the RULES table as Ruby for parsing grammars:
41
54
 
42
- ebnf.make_bnf
55
+ grammar.make_peg
56
+ grammar.to_ruby
43
57
 
44
- Generate [First/Follow][] rules for BNF grammars
58
+ Transform [EBNF][] to [BNF][] (generates sub-rules using `alt` or `seq` from `plus`, `star` or `opt`)
45
59
 
46
- ebnf.first_follow(:ebnf)
60
+ grammar.make_bnf
47
61
 
48
- Generate Terminal, [First/Follow][], Cleanup and Branch tables as Ruby for parsing grammars
62
+ Generate [First/Follow][] rules for BNF grammars (using "ebnf" as the starting production):
49
63
 
50
- ebnf.build_tables
51
- ebnf.to_ruby
64
+ grammar.first_follow(:ebnf)
52
65
 
53
- Generate formatted grammar using HTML (requires [Haml][Haml] gem)
66
+ Generate Terminal, [First/Follow][], Cleanup and Branch tables as Ruby for parsing grammars:
54
67
 
55
- ebnf.to_html
68
+ grammar.build_tables
69
+ grammar.to_ruby
56
70
 
57
- ### Parser S-Expressions
58
- Intermediate representations of the grammar may be serialized to Lisp-like S-Expressions. For example, the rule `[1] ebnf ::= (declaration | rule)*` is serialized as `(rule ebnf "1" (star (alt declaration rule)))`.
59
-
60
- Once the [LL(1)][] conversion is made, the [First/Follow][] table is generated, this rule expands as follows:
61
-
62
- (rule ebnf "1"
63
- (start #t)
64
- (first "@pass" "@terminals" LHS _eps)
65
- (follow _eof)
66
- (cleanup star)
67
- (alt _empty _ebnf_2))
68
- (rule _ebnf_1 "1.1"
69
- (first "@pass" "@terminals" LHS)
70
- (follow "@pass" "@terminals" LHS _eof)
71
- (alt declaration rule))
72
- (rule _ebnf_2 "1.2"
73
- (first "@pass" "@terminals" LHS)
74
- (follow _eof)
75
- (cleanup merge)
76
- (seq _ebnf_1 ebnf))
77
- (rule _ebnf_3 "1.3" (first "@pass" "@terminals" LHS _eps) (follow _eof) (seq ebnf))
78
-
79
- ### Creating terminal definitions and parser rules to parse generated grammars
80
- The parser is initialized to callbacks invoked on entry and exit
81
- to each `terminal` and `production`. A trivial parser loop can be described as follows:
82
-
83
- require 'ebnf/ll1/parser'
84
- require 'meta'
85
-
86
- class Parser
87
- include Meta
88
-
89
- terminal(:SYMBOL, /([a-z]|[A-Z]|[0-9]|_)+/) do |prod, token, input|
90
- # Add data based on scanned token to input
91
- input[:symbol] = token.value
92
- end
93
-
94
- start_production(:rule) do |input, current, callback|
95
- # Process on start of production
96
- # Set state for entry into recursed rules through current
97
-
98
- # Callback to parser loop with callback
99
- end
100
-
101
- production(:rule) do |input, current, callback|
102
- # Process on end of production
103
- # return results in input, retrieve results from recursed rules in current
104
-
105
- # Callback to parser loop with callback
106
- end
107
-
108
- def initialize(input)
109
- parser_options = {
110
- branch: BRANCH,
111
- first: FIRST,
112
- follow: FOLLOW,
113
- cleanup: CLEANUP
114
- }
115
- parse(input, start_symbol, parser_options) do |context, *data|
116
- # Process calls from callback from productions
117
-
118
- rescue ArgumentError, RDF::LL1::Parser::Error => e
119
- progress("Parsing completed with errors:\n\t#{e.message}")
120
- raise RDF::ReaderError, e.message if validate?
121
- end
122
-
123
- ### Branch Table
124
- The Branch table is a hash mapping production rules to a hash relating terminals appearing in input to sequence of productions to follow when the corresponding input terminal is found. This allows either the `seq` primitive, where all terminals map to the same sequence of productions, or the `alt` primitive, where each terminal may map to a different production.
125
-
126
- BRANCH = {
127
- :alt => {
128
- "(" => [:seq, :_alt_1],
129
- :ENUM => [:seq, :_alt_1],
130
- :HEX => [:seq, :_alt_1],
131
- :O_ENUM => [:seq, :_alt_1],
132
- :O_RANGE => [:seq, :_alt_1],
133
- :RANGE => [:seq, :_alt_1],
134
- :STRING1 => [:seq, :_alt_1],
135
- :STRING2 => [:seq, :_alt_1],
136
- :SYMBOL => [:seq, :_alt_1],
137
- },
138
- ...
139
- :declaration => {
140
- "@pass" => [:pass],
141
- "@terminals" => ["@terminals"],
142
- },
143
- ...
144
- }
145
-
146
- In this case the `alt` rule is `seq ('|' seq)*` can happen when any of the specified tokens appears on the input stream. The all cause the same token to be passed to the `seq` rule and follow with `_alt_1`, which handles the `('|' seq)*` portion of the rule, after the first sequence is matched.
147
-
148
- The `declaration` rule is `@terminals' | pass` using the `alt` primitive determining the production to run based on the terminal appearing on the input stream. Eventually, a terminal production is found and the token is consumed.
149
-
150
- ### First/Follow Table
151
- The [First/Follow][] table is a hash mapping production rules to the terminals that may proceed or follow the rule. For example:
152
-
153
- FIRST = {
154
- :alt => [
155
- :HEX,
156
- :SYMBOL,
157
- :ENUM,
158
- :O_ENUM,
159
- :RANGE,
160
- :O_RANGE,
161
- :STRING1,
162
- :STRING2,
163
- "("],
164
- ...
165
- }
166
-
167
- ### Terminals Table
168
- This table is a simple list of the terminal productions found in the grammar. For example:
169
-
170
- TERMINALS = ["(", ")", "-",
171
- "@pass", "@terminals",
172
- :ENUM, :HEX, :LHS, :O_ENUM, :O_RANGE,:POSTFIX,
173
- :RANGE, :STRING1, :STRING2, :SYMBOL,"|"
174
- ].freeze
175
-
176
- ### Cleanup Table
177
- This table identifies productions which used EBNF rules, which are transformed to BNF for actual parsing. This allows the parser, in some cases, to reproduce *star*, *plus*, and *opt* rule matches. For example:
178
-
179
- CLEANUP = {
180
- :_alt_1 => :star,
181
- :_alt_3 => :merge,
182
- :_diff_1 => :opt,
183
- :ebnf => :star,
184
- :_ebnf_2 => :merge,
185
- :_postfix_1 => :opt,
186
- :seq => :plus,
187
- :_seq_1 => :star,
188
- :_seq_2 => :merge,
189
- }.freeze
190
-
191
- In this case the `ebnf` rule was `(declaration | rule)*`. As BNF does not support a star operator, this is decomposed into a set of rules using `alt` and `seq` primitives:
192
-
193
- ebnf ::= _empty _ebnf_2
194
- _ebnf_1 ::= declaration | rule
195
- _ebnf_2 ::= _ebnf_1 ebnf
196
- _ebnf_3 ::= ebnf
197
-
198
- The `_empty` production matches an empty string, so allows for now value. `_ebnf_2` matches `declaration | rule` (using the `alt` primitive) followed by `ebnf`, creating a sequence of zero or more `declaration` or `alt` members.
71
+ Generate formatted grammar using HTML (requires [Haml][Haml] gem):
199
72
 
200
- ## EBNF Grammar
201
- The [EBNF][] variant used here is based on [W3C](http://w3.org/) [EBNF][] (see {file:etc/ebnf.ebnf EBNF grammar}) as defined in the
202
- [XML 1.0 recommendation](http://www.w3.org/TR/REC-xml/), with minor extensions:
73
+ grammar.to_html
74
+
75
+ ### Parsing an ISO/IEC 14977 Grammar
76
+
77
+ The EBNF gem can also parse [ISO/EIC 14977] Grammars (ISOEBNF) to [S-Expressions][S-Expression].
78
+
79
+ grammar = EBNF.parse(File.open('./etc/iso-ebnf.isoebnf', format: :isoebnf))
80
+
81
+ ### Parsing an ABNF Grammar
82
+
83
+ The EBNF gem can also parse [ABNF] Grammars to [S-Expressions][S-Expression].
84
+
85
+ grammar = EBNF.parse(File.open('./etc/abnf.abnf', format: :abnf))
86
+
87
+ ### Parser Debugging
88
+
89
+ Inevitably while implementing a parser for some specific grammar, a developer will need greater insight into the operation of the parser. While this can involve sorting through a tremendous amount of data, the parser can be provided a [Logger][] instance which will output messages at varying levels of detail to document the state of the parser at any given point. Most useful is likely the `INFO` level of debugging, but even more detail is revealed using the `DEBUG` level. `WARN` and `ERROR` statements will typically also be provided as part of an exception if parsing fails, but can be shown in the context of other parsing state with appropriate indentation as part of the logger.
90
+
91
+ ### Writing Grammars
92
+
93
+ The {EBNF::Writer} class can be used to write parsed grammars out, either as formatted text, or HTML. Because grammars are written from the Abstract Syntax Tree, represented as [S-Expressions][S-Expression], this provides a means of transforming between grammar formats (e.g., W3C [EBNF][] to [ABNF][]), although with some potential loss in semantic fidelity (case-insensitive string matching vs. case-sensitive matching).
94
+
95
+ The formatted HTML results are designed to be appropriate for including in specifications.
96
+
97
+ ### Parser Errors
98
+ On a parsing failure, and exception is raised with information that may be useful in determining the source of the error.
203
99
 
204
- * Comments include `\\` and `#` through end of line (other than hex character) and `/* ... */ (* ... *) which may cross lines`
100
+ ## EBNF Grammar
101
+ The [EBNF][] variant used here is based on [W3C](https://w3.org/) [EBNF][] (see {file:etc/ebnf.ebnf EBNF grammar}) as defined in the
102
+ [XML 1.0 recommendation](https://www.w3.org/TR/REC-xml/), with minor extensions:
103
+
104
+ Note that the grammar includes an optional `[identifer]` in front of rule names, which can be in conflict with the `RANGE` terminal. It is typically not a problem, but if it comes up, try parsing with the `native` parser, add comments or sequences to disambiguate. EBNF does not have beginning of line checks as all whitespace is treated the same, so the common practice of identifying each rule inherently leads to such ambiguity.
105
+
106
+ The character set for EBNF is UTF-8.
107
+
108
+ The general form of a rule is:
109
+
110
+ symbol ::= expression
111
+
112
+ which can also be proceeded by an optional number enclosed in square brackets to identify the rule number:
113
+
114
+ [1] symbol ::= expression
115
+
116
+ (Note, this can introduce an ambiguity if the previous rule ends in a range or enum and the current rule has no identifier. In this case, enclosing `expression` within parentheses, or adding intervening comments can resolve the ambiguity.)
117
+
118
+ Symbols are written in CAPITAL CASE if they are the start symbol of a regular language (terminals), otherwise with they are treated as non-terminal rules. Literal strings are quoted.
119
+
120
+ Within the expression on the right-hand side of a rule, the following expressions are used to match strings of one or more characters:
121
+
122
+ <table>
123
+ <tr><td><code>#xN</code></td>
124
+ <td>where <code>N</code> is a hexadecimal integer, the expression matches the character whose number (code point) in ISO/IEC 10646 is <code>N</code>. The number of leading zeros in the <code>#xN</code> form is insignificant.</td></tr>
125
+ <tr><td><code>[a-zA-Z], [#xN-#xN]</code>
126
+ <td>matches any Char or HEX with a value in the range(s) indicated (inclusive).</td></tr>
127
+ <tr><td><code>[abc], [#xN#xN#xN]</code></td>
128
+ <td>matches any UTF-8 R\_CHAR or HEX with a value among the characters enumerated. The last component may be '-'. Enumerations and ranges may be mixed in one set of brackets.</td></tr>
129
+ <tr><td><code>[^a-z], [^#xN-#xN]</code></td>
130
+ <td>matches any UTF-8 Char or HEX a value outside the range indicated.</td></tr>
131
+ <tr><td><code>[^abc], [^#xN#xN#xN]</code></td>
132
+ <td>matches any UTF-8 R\_CHAR or HEX with a value not among the characters given. The last component may be '-'. Enumerations and ranges of excluded values may be mixed in one set of brackets.</td></tr>
133
+ <tr><td><code>"string"</code></td>
134
+ <td>matches a literal string matching that given inside the double quotes.</td></tr>
135
+ <tr><td><code>'string'</code></td>
136
+ <td>matches a literal string matching that given inside the single quotes.</td></tr>
137
+ <tr><td><code>A (B | C)</code></td>
138
+ <td><code>(B | C)</code> is treated as a unit and may be combined as described in this list.</td></tr>
139
+ <tr><td><code>A?</code></td>
140
+ <td>matches A or nothing; optional A.</td></tr>
141
+ <tr><td><code>A B</code></td>
142
+ <td>matches <code>A</code> followed by <code>B</code>. This operator has higher precedence than alternation; thus <code>A B | C D</code> is identical to <code>(A B) | (C D)</code>.</td></tr>
143
+ <tr><td><code>A | B</code></td>
144
+ <td>matches <code>A</code> or <code>B</code>.</td></tr>
145
+ <tr><td><code>A - B</code></td>
146
+ <td>matches any string that matches <code>A</code> but does not match <code>B</code>. (Only supported on Terminals in LL(1) BNF).</td></tr>
147
+ <tr><td><code>A+</code></td>
148
+ <td>matches one or more occurrences of <code>A</code>. Concatenation has higher precedence than alternation; thus <code>A+ | B+</code> is identical to <code>(A+) | (B+)</code>.</td></tr>
149
+ <tr><td><code>A*</code></td>
150
+ <td>matches zero or more occurrences of <code>A</code>. Concatenation has higher precedence than alternation; thus <code>A* | B*</code> is identical to <code>(A*) | (B*)</code>.</td></tr>
151
+ <tr><td><code>@pass " "*</code></td>
152
+ <td>Defines consumed whitespace in the document. Any whitespace found between non-terminal rules is consumed and ignored.</td></tr>
153
+ <tr><td><code>@terminals</code></td>
154
+ <td>Introduces terminal rules. All rules defined after this point are treated as terminals.</td></tr>
155
+ </table>
156
+
157
+ * Comments include `//` and `#` through end of line (other than hex character) and `/* ... */ (* ... *) which may cross lines`
205
158
  * All rules **MAY** start with an identifier, contained within square brackets. For example `[1] rule`, where the value within the brackets is a symbol `([a-z] | [A-Z] | [0-9] | "_" | ".")+`
206
- * `@terminals` causes following rules to be treated as terminals. Any terminal which are entirely upper-case are also treated as terminals
159
+ * `@terminals` causes following rules to be treated as terminals. Any terminal which is all upper-case (eg`TERMINAL`), or any rules with expressions that match characters (`#xN`, `[a-z]`, `[^a-z]`, `[abc]`, `[^abc]`, `"string"`, `'string'`, or `A - B`), are also treated as terminals.
207
160
  * `@pass` defines the expression used to detect whitespace, which is removed in processing.
208
161
  * No support for `wfc` (well-formedness constraint) or `vc` (validity constraint).
209
162
 
210
- Parsing this grammar yields an S-Expression version: {file:etc/ebnf.ll1.sxp}.
163
+ Parsing this grammar yields an [S-Expression][] version: {file:etc/ebnf.sxp} (or [LL(1)][] version {file:etc/ebnf.ll1.sxp} or [PEG][] version {file:etc/ebnf.peg.sxp}).
164
+
165
+ ### Parser S-Expressions
166
+ Intermediate representations of the grammar may be serialized to Lisp-like [S-Expressions][S-Expression]. For example, the rule
167
+
168
+ [1] ebnf ::= (declaration | rule)*
169
+
170
+ is serialized as
171
+
172
+ (rule ebnf "1" (star (alt declaration rule)))
173
+
174
+ Different components of an EBNF rule expression are transformed into their own operator:
175
+
176
+ <table>
177
+ <tr><td><code>#xN</code></td><td><code>(hex "#xN")</code></td></tr>
178
+ <tr><td><code>[a-z#xN-#xN]</code></td><td><code>(range "a-z#xN-#xN")</code></td></tr>
179
+ <tr><td><code>[abc#xN]</code></td><td><code>(range "abc#xN")</code></td></tr>
180
+ <tr><td><code>[^a-z#xN-#xN]</code></td><td><code>(range "^a-z#xN-#xN")</code></td></tr>
181
+ <tr><td><code>[^abc#xN]</code></td><td><code>(range "^abc#xN")</code></td></tr>
182
+ <tr><td><code>"string"</code></td><td><code>"string"</code></td></tr>
183
+ <tr><td><code>'string'</code></td><td><code>"string"</code></td></tr>
184
+ <tr><td><code>A (B | C)</code></td><td><code>(seq (A (alt B C)))</code></td></tr>
185
+ <tr><td><code>A?</code></td><td><code>(opt A)</code></td></tr>
186
+ <tr><td><code>A B</code></td><td><code>(seq A B)</code></td></tr>
187
+ <tr><td><code>A | B</code></td><td><code>(alt A B)</code></td></tr>
188
+ <tr><td><code>A - B</code></td>
189
+ <td><code>(diff A B) for terminals.<br/>
190
+ <code>(seq (not B) A) for non-terminals (PEG parsing only)</code></code></td></tr>
191
+ <tr><td><code>A+</code></td><td><code>(plus A)</code></td></tr>
192
+ <tr><td><code>A*</code></td><td><code>(star A)</code></td></tr>
193
+ <tr><td><code>@pass " "*</code></td><td><code>(pass _pass (star " "))</code></td></tr>
194
+ <tr><td><code>@terminals</code></td><td></td></tr>
195
+ </table>
196
+
197
+ Other rule operators are not directly supported in [EBNF][], but are included to support other notations (e.g., [ABNF][] and [ISO/IEC 14977][]):
198
+
199
+ <table>
200
+ <tr><td><code>%i"StRiNg"</code></td><td><code>(istr "StRiNg")</code></td><td>Case-insensitive string matching</td></tr>
201
+ <tr><td><code>'' - A</code></td><td><code>(not A)</code></td><td>Negative look-ahead, used for non-terminal uses of `B - A`.</td></tr>
202
+ <tr><td><code>n*mA</code></td><td><code>(rept n m A)</code></td><td>Explicit repetition.</td></tr>
203
+ </table>
204
+
205
+ Additionally, rules defined with an UPPERCASE symbol are treated as terminals.
206
+
207
+ For an [LL(1)][] parser generator, the {EBNF::BNF.make_bnf} method can be used to transform the EBNF rule into a BNF rule.
208
+
209
+ (rule ebnf "1" (alt _empty _ebnf_2))
210
+ (rule _ebnf_1 "1.1" (alt declaration rule))
211
+ (rule _ebnf_2 "1.2" (seq _ebnf_1 ebnf))
212
+ (rule _ebnf_3 "1.3" (seq ebnf))
213
+
214
+ This allows [First/Follow][] and other tables used by a parser to parse examples of the associated grammar. For more, see {EBNF::LL1}.
215
+
216
+ For a [PEG][] parser generator, there is a simpler transformation that reduces rules containing sub-expressions (composed of `star`, `alt`, `seq` and similar expressions) and creates named rules to allow appropriate callbacks and for naming elements of the generating abstract syntax tree. The {EBNF::PEG.make_peg} method transforms the original rule into the following two rules:
217
+
218
+ (rule ebnf "1" (star _ebnf_1))
219
+ (rule _ebnf_1 "1.1" (alt declaration rule))
220
+
221
+ ## Example parsers
222
+ For a [PEG][] parser for a simple grammar implementing a calculator see [Calc example](https://dryruby.github.io/ebnf/examples/calc/doc/calc.html)
223
+
224
+ For an example parser built using this gem that parses the [EBNF][] grammar, see [EBNF PEG Parser example](https://dryruby.github.io/ebnf/examples/ebnf-peg-parser/doc/parser.html). This example creates a parser for the [EBNF][] grammar which generates the same Abstract Syntax Tree as the built-in parser in the gem.
225
+
226
+ There is also an
227
+ [EBNF LL(1) Parser example](https://dryruby.github.io/ebnf/examples/ebnf-peg-parser/doc/parser.html).
228
+
229
+ The [ISO EBNF Parser](https://dryruby.github.io/ebnf/examples/isoebnf/doc/parser.html) example parses [ISO/IEC 14977][] into [S-Expressions][S-Expression], which can be used to parse compatible grammars using this parser (either [PEG][] or [LL(1)][]).
211
230
 
212
- ## Example parser
213
- For an example parser built using this gem, see {file:examples/ebnf-parser/README EBNF Parser example}. This example creates a parser for the [EBNF][] grammar which generates the same Abstract Syntax Tree as the built-in parser in the gem.
231
+ The [ABNF Parser](https://dryruby.github.io/ebnf/examples/abnf/doc/parser.html) example parses [ABNF][] into [S-Expressions][S-Expression], which can be used to parse compatible grammars using this [PEG][] parser.
214
232
 
215
233
  ## Acknowledgements
216
234
  Much of this work, particularly the generic parser, is inspired by work originally done by
217
- Tim Berners-Lee's Python [predictive parser](http://www.w3.org/2000/10/swap/grammar/predictiveParser.py).
235
+ Tim Berners-Lee's Python [predictive parser](https://www.w3.org/2000/10/swap/grammar/predictiveParser.py).
218
236
 
219
- The EBNF parser was inspired by Dan Connolly's
220
- [EBNF to Turtle processor](http://www.w3.org/2000/10/swap/grammar/ebnf2turtle.py),
221
- [EBNF to BNF Notation-3 rules](http://www.w3.org/2000/10/swap/grammar/ebnf2bnf.n3),
222
- and [First Follow Notation-3 rules](http://www.w3.org/2000/10/swap/grammar/first_follow.n3).
237
+ The [LL(1)][] parser was inspired by Dan Connolly's
238
+ [EBNF to Turtle processor](https://www.w3.org/2000/10/swap/grammar/ebnf2turtle.py),
239
+ [EBNF to BNF Notation-3 rules](https://www.w3.org/2000/10/swap/grammar/ebnf2bnf.n3),
240
+ and [First Follow Notation-3 rules](https://www.w3.org/2000/10/swap/grammar/first_follow.n3).
223
241
 
224
242
  ## Documentation
225
243
  Full documentation available on [Rubydoc.info][EBNF doc].
226
244
 
227
245
  ## Future Work
228
246
  * Better LL(1) parser tests
229
- * Either generate [Packrat parser][Packrat] for a [Parsing Regular Expression Grammar][PEG], or integrate with [Treetop][] or similar.
230
247
 
231
248
  ## Author
232
- * [Gregg Kellogg](http://github.com/gkellogg) - <http://greggkellogg.net/>
249
+ * [Gregg Kellogg](https://github.com/gkellogg) - <https://greggkellogg.net/>
233
250
 
234
251
  ## Contributing
235
252
  This repository uses [Git Flow](https://github.com/nvie/gitflow) to mange development and release activity. All submissions _must_ be on a feature branch based on the _develop_ branch to ease staging and integration.
@@ -244,26 +261,32 @@ This repository uses [Git Flow](https://github.com/nvie/gitflow) to mange develo
244
261
  list in the the `README`. Alphabetical order applies.
245
262
  * Do note that in order for us to merge any non-trivial changes (as a rule
246
263
  of thumb, additions larger than about 15 lines of code), we need an
247
- explicit [public domain dedication][PDD] on record from you.
264
+ explicit [public domain dedication][PDD] on record from you,
265
+ which you will be asked to agree to on the first commit to a repo within the organization.
248
266
 
249
267
  ## License
250
268
  This is free and unencumbered public domain software. For more information,
251
- see <http://unlicense.org/> or the accompanying {file:UNLICENSE} file.
252
-
253
- A copy of the [Turtle EBNF][] and derived parser files are included in the repository, which are not covered under the UNLICENSE. These files are covered via the [W3C Document License](http://www.w3.org/Consortium/Legal/2002/copyright-documents-20021231).
254
-
255
- [Ruby]: http://ruby-lang.org/
256
- [YARD]: http://yardoc.org/
257
- [YARD-GS]: http://rubydoc.info/docs/yard/file/docs/GettingStarted.md
258
- [PDD]: http://lists.w3.org/Archives/Public/public-rdf-ruby/2010May/0013.html
259
- [EBNF]: http://www.w3.org/TR/REC-xml/#sec-notation
260
- [EBNF doc]: http://rubydoc.info/github/dryruby/ebnf/master/frames
261
- [First/Follow]: http://en.wikipedia.org/wiki/LL_parser#Constructing_an_LL.281.29_parsing_table
262
- [LL(1)]: http://www.csd.uwo.ca/~moreno//CS447/Lectures/Syntax.html/node14.html
263
- [LL(1) Parser]: http://en.wikipedia.org/wiki/LL_parser
264
- [Tokenizer]: http://en.wikipedia.org/wiki/Lexical_analysis#Tokenizer
265
- [Turtle EBNF]: http://dvcs.w3.org/hg/rdf/file/default/rdf-turtle/turtle.bnf
266
- [Packrat]: http://pdos.csail.mit.edu/~baford/packrat/thesis/
267
- [PEG]: http://en.wikipedia.org/wiki/Parsing_expression_grammar
268
- [Treetop]: http://rubygems.org/gems/treetop
269
- [Haml]: http://rubygems.org/gems/haml
269
+ see <https://unlicense.org/> or the accompanying {file:UNLICENSE} file.
270
+
271
+ A copy of the [Turtle EBNF][] and derived parser files are included in the repository, which are not covered under the UNLICENSE. These files are covered via the [W3C Document License](https://www.w3.org/Consortium/Legal/2002/copyright-documents-20021231).
272
+
273
+ [Ruby]: https://ruby-lang.org/
274
+ [YARD]: https://yardoc.org/
275
+ [YARD-GS]: https://rubydoc.info/docs/yard/file/docs/GettingStarted.md
276
+ [PDD]: https://lists.w3.org/Archives/Public/public-rdf-ruby/2010May/0013.html
277
+ [ABNF]: https://www.rfc-editor.org/rfc/rfc5234
278
+ [BNF]: https://en.wikipedia.org/wiki/Backus–Naur_form
279
+ [EBNF]: https://www.w3.org/TR/REC-xml/#sec-notation
280
+ [EBNF doc]: https://rubydoc.info/github/dryruby/ebnf
281
+ [First/Follow]: https://en.wikipedia.org/wiki/LL_parser#Constructing_an_LL.281.29_parsing_table
282
+ [ISO/IEC 14977]:https://www.iso.org/standard/26153.html
283
+ [LL(1)]: https://www.csd.uwo.ca/~moreno//CS447/Lectures/Syntax.html/node14.html
284
+ [LL(1) Parser]: https://en.wikipedia.org/wiki/LL_parser
285
+ [Logger]: https://ruby-doc.org/stdlib-2.4.0/libdoc/logger/rdoc/Logger.html
286
+ [S-expression]: https://en.wikipedia.org/wiki/S-expression
287
+ [Tokenizer]: https://en.wikipedia.org/wiki/Lexical_analysis#Tokenizer
288
+ [Turtle]: https://www.w3.org/TR/2012/WD-turtle-20120710/
289
+ [Turtle EBNF]: https://dvcs.w3.org/hg/rdf/file/default/rdf-turtle/turtle.bnf
290
+ [Packrat]: https://pdos.csail.mit.edu/~baford/packrat/thesis/
291
+ [PEG]: https://en.wikipedia.org/wiki/Parsing_expression_grammar
292
+ [Haml]: https://rubygems.org/gems/haml