treetop 1.4.8 → 1.4.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/Rakefile +15 -0
- data/doc/contributing_and_planned_features.markdown +0 -1
- data/doc/index.markdown +19 -21
- data/doc/semantic_interpretation.markdown +30 -1
- data/doc/site.rb +7 -7
- data/doc/site/contribute.html +124 -0
- data/doc/site/images/bottom_background.png +0 -0
- data/doc/site/images/middle_background.png +0 -0
- data/doc/site/images/paren_language_output.png +0 -0
- data/doc/site/images/pivotal.gif +0 -0
- data/doc/site/images/top_background.png +0 -0
- data/doc/site/index.html +102 -0
- data/doc/site/pitfalls_and_advanced_techniques.html +68 -0
- data/doc/site/robots.txt +5 -0
- data/doc/site/screen.css +134 -0
- data/doc/site/semantic_interpretation.html +245 -0
- data/doc/site/syntactic_recognition.html +271 -0
- data/doc/site/using_in_ruby.html +123 -0
- data/doc/sitegen.rb +1 -1
- data/doc/syntactic_recognition.markdown +109 -3
- data/doc/using_in_ruby.markdown +79 -2
- data/examples/lambda_calculus/arithmetic.treetop +13 -12
- data/examples/lambda_calculus/arithmetic_node_classes.rb +4 -2
- data/examples/lambda_calculus/arithmetic_test.rb +4 -0
- data/lib/treetop/runtime/compiled_parser.rb +3 -2
- data/lib/treetop/version.rb +1 -1
- data/spec/runtime/compiled_parser_spec.rb +31 -9
- metadata +20 -3
data/doc/using_in_ruby.markdown
CHANGED
@@ -1,6 +1,6 @@
|
|
1
1
|
#Using Treetop Grammars in Ruby
|
2
2
|
##Using the Command Line Compiler
|
3
|
-
You can `.treetop` files into Ruby source code with the `tt` command line script. `tt` takes an list of files with a `.treetop` extension and compiles them into `.rb` files of the same name. You can then `require` these files like any other Ruby script. Alternately, you can supply just one `.treetop` file and a `-o` flag to name specify the name of the output file. Improvements to this compilation script are welcome.
|
3
|
+
You can compile `.treetop` files into Ruby source code with the `tt` command line script. `tt` takes an list of files with a `.treetop` extension and compiles them into `.rb` files of the same name. You can then `require` these files like any other Ruby script. Alternately, you can supply just one `.treetop` file and a `-o` flag to name specify the name of the output file. Improvements to this compilation script are welcome.
|
4
4
|
|
5
5
|
tt foo.treetop bar.treetop
|
6
6
|
tt foo.treetop -o foogrammar.rb
|
@@ -16,7 +16,10 @@ In order to use Polyglot dynamic loading of `.treetop` or `.tt` files though, yo
|
|
16
16
|
in order to use Polyglot auto loading with Treetop in Ruby.
|
17
17
|
|
18
18
|
##Instantiating and Using Parsers
|
19
|
-
If a grammar by the name of `Foo` is defined, the compiled Ruby source will define a `FooParser` class.
|
19
|
+
If a grammar by the name of `Foo` is defined, the compiled Ruby source will define a `FooParser` class.
|
20
|
+
To parse input, create an instance and call its `parse` method with a string.
|
21
|
+
The parser will return the syntax tree of the match or `nil` if there is a failure.
|
22
|
+
Note that by default, the parser will fail unless *all* input is consumed.
|
20
23
|
|
21
24
|
Treetop.load "arithmetic"
|
22
25
|
|
@@ -26,3 +29,77 @@ If a grammar by the name of `Foo` is defined, the compiled Ruby source will defi
|
|
26
29
|
else
|
27
30
|
puts 'failure'
|
28
31
|
end
|
32
|
+
|
33
|
+
##Parser Options
|
34
|
+
A Treetop parser has several options you may set.
|
35
|
+
Some are settable permanently by methods on the parser, but all may be passed in as options to the `parse` method.
|
36
|
+
|
37
|
+
parser = ArithmeticParser.new
|
38
|
+
input = 'x = 2; y = x+3;'
|
39
|
+
|
40
|
+
# Temporarily override an option:
|
41
|
+
result1 = parser.parse(input, :consume_all_input => false)
|
42
|
+
puts "consumed #{parser.index} characters"
|
43
|
+
|
44
|
+
parser.consume_all_input = false
|
45
|
+
result1 = parser.parse(input)
|
46
|
+
puts "consumed #{parser.index} characters"
|
47
|
+
|
48
|
+
# Continue the parse with the next character:
|
49
|
+
result2 = parser.parse(input, :index => parser.index)
|
50
|
+
|
51
|
+
# Parse, but match rule `variable` instead of the normal root rule:
|
52
|
+
parser.parse(input, :root => :variable)
|
53
|
+
parser.root = :variable # Permanent setting
|
54
|
+
|
55
|
+
If you have a statement-oriented language, you can save memory by parsing just one statement at a time,
|
56
|
+
and discarding the parse tree after each statement.
|
57
|
+
|
58
|
+
|
59
|
+
##Learning From Failure
|
60
|
+
If a parse fails, it returns nil. In this case, you can ask the parser for an explanation.
|
61
|
+
The failure reasons include the terminal nodes which were tried at the furthermost point the parse reached.
|
62
|
+
|
63
|
+
parser = ArithmeticParser.new
|
64
|
+
result = parser.parse('4+=3')
|
65
|
+
|
66
|
+
if !result
|
67
|
+
puts parser.failure_reason
|
68
|
+
puts parser.failure_line
|
69
|
+
puts parser.failure_column
|
70
|
+
end
|
71
|
+
|
72
|
+
=>
|
73
|
+
Expected one of (, - at line 1, column 3 (byte 3) after +
|
74
|
+
1
|
75
|
+
3
|
76
|
+
|
77
|
+
|
78
|
+
##Using Parse Results
|
79
|
+
Please don't try to walk down the syntax tree yourself, and please don't use the tree as your own convenient data structure.
|
80
|
+
It contains many more nodes than your application needs, often even more than one for every character of input.
|
81
|
+
|
82
|
+
parser = ArithmeticParser.new
|
83
|
+
p parser.parse('2+3')
|
84
|
+
|
85
|
+
=>
|
86
|
+
SyntaxNode+Additive1 offset=0, "2+3" (multitive):
|
87
|
+
SyntaxNode+Multitive1 offset=0, "2" (primary):
|
88
|
+
SyntaxNode+Number0 offset=0, "2":
|
89
|
+
SyntaxNode offset=0, ""
|
90
|
+
SyntaxNode offset=0, "2"
|
91
|
+
SyntaxNode offset=1, ""
|
92
|
+
SyntaxNode offset=1, ""
|
93
|
+
SyntaxNode offset=1, "+3":
|
94
|
+
SyntaxNode+Additive0 offset=1, "+3" (multitive):
|
95
|
+
SyntaxNode offset=1, "+"
|
96
|
+
SyntaxNode+Multitive1 offset=2, "3" (primary):
|
97
|
+
SyntaxNode+Number0 offset=2, "3":
|
98
|
+
SyntaxNode offset=2, ""
|
99
|
+
SyntaxNode offset=2, "3"
|
100
|
+
SyntaxNode offset=3, ""
|
101
|
+
SyntaxNode offset=3, ""
|
102
|
+
|
103
|
+
Instead, add methods to the root rule which return the information you require in a sensible form.
|
104
|
+
Each rule can call its sub-rules, and this method of walking the syntax tree is much preferable to
|
105
|
+
attempting to walk it from the outside.
|
@@ -4,7 +4,10 @@ grammar Arithmetic
|
|
4
4
|
end
|
5
5
|
|
6
6
|
rule comparative
|
7
|
-
|
7
|
+
head:additive
|
8
|
+
tail:(
|
9
|
+
space operator:equality_op
|
10
|
+
space operand:additive)* <BinaryOperation>
|
8
11
|
end
|
9
12
|
|
10
13
|
rule equality_op
|
@@ -16,11 +19,10 @@ grammar Arithmetic
|
|
16
19
|
end
|
17
20
|
|
18
21
|
rule additive
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
multitive
|
22
|
+
head:multitive
|
23
|
+
tail:(
|
24
|
+
space operator:additive_op
|
25
|
+
space operand:multitive)* <BinaryOperation>
|
24
26
|
end
|
25
27
|
|
26
28
|
rule additive_op
|
@@ -38,11 +40,10 @@ grammar Arithmetic
|
|
38
40
|
end
|
39
41
|
|
40
42
|
rule multitive
|
41
|
-
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
primary
|
43
|
+
head:primary
|
44
|
+
tail:(
|
45
|
+
space operator:multitive_op
|
46
|
+
space operand:primary)* <BinaryOperation>
|
46
47
|
end
|
47
48
|
|
48
49
|
rule multitive_op
|
@@ -94,4 +95,4 @@ grammar Arithmetic
|
|
94
95
|
rule space
|
95
96
|
' '*
|
96
97
|
end
|
97
|
-
end
|
98
|
+
end
|
@@ -1,7 +1,9 @@
|
|
1
1
|
module Arithmetic
|
2
2
|
class BinaryOperation < Treetop::Runtime::SyntaxNode
|
3
3
|
def eval(env={})
|
4
|
-
|
4
|
+
tail.elements.inject(head.eval(env)) do |value, element|
|
5
|
+
element.operator.apply(value, element.operand.eval(env))
|
6
|
+
end
|
5
7
|
end
|
6
8
|
end
|
7
|
-
end
|
9
|
+
end
|
@@ -43,6 +43,10 @@ class ArithmeticParserTest < Test::Unit::TestCase
|
|
43
43
|
assert_equal 11, parse('1 + 2 * 3 + 4').eval
|
44
44
|
end
|
45
45
|
|
46
|
+
def test_left_to_right
|
47
|
+
assert_equal 2, parse('5 - 2 - 1').eval
|
48
|
+
end
|
49
|
+
|
46
50
|
def test_parentheses
|
47
51
|
assert_equal 25, parse('(5 + x) * (10 - y)').eval('x' => 0, 'y' => 5)
|
48
52
|
end
|
@@ -15,8 +15,9 @@ module Treetop
|
|
15
15
|
def parse(input, options = {})
|
16
16
|
prepare_to_parse(input)
|
17
17
|
@index = options[:index] if options[:index]
|
18
|
-
result = send("_nt_#{root}")
|
19
|
-
|
18
|
+
result = send("_nt_#{options[:root] || root}")
|
19
|
+
should_consume_all = options.include?(:consume_all_input) ? options[:consume_all_input] : consume_all_input?
|
20
|
+
return nil if (should_consume_all && index != input.size)
|
20
21
|
return SyntaxNode.new(input, index...(index + 1)) if result == true
|
21
22
|
return result
|
22
23
|
end
|
data/lib/treetop/version.rb
CHANGED
@@ -3,45 +3,67 @@ require 'spec_helper'
|
|
3
3
|
module CompiledParserSpec
|
4
4
|
describe Runtime::CompiledParser, "for a grammar with two rules" do
|
5
5
|
attr_reader :parser
|
6
|
-
|
6
|
+
|
7
7
|
testing_grammar %{
|
8
8
|
grammar TwoRules
|
9
9
|
rule a
|
10
10
|
'a'
|
11
11
|
end
|
12
|
-
|
12
|
+
|
13
13
|
rule b
|
14
14
|
'b'
|
15
15
|
end
|
16
16
|
end
|
17
17
|
}
|
18
|
-
|
18
|
+
|
19
19
|
before do
|
20
20
|
@parser = parser_class_under_test.new
|
21
21
|
end
|
22
|
-
|
22
|
+
|
23
23
|
it "allows its root to be specified" do
|
24
24
|
parser.parse('a').should_not be_nil
|
25
25
|
parser.parse('b').should be_nil
|
26
|
-
|
26
|
+
|
27
|
+
# Check that the temporary-override works:
|
28
|
+
parser.parse('b', :root => :b).should_not be_nil
|
29
|
+
parser.parse('a', :root => :b).should be_nil
|
30
|
+
|
31
|
+
# Check that the temporary-override isn't sticky:
|
32
|
+
parser.parse('a').should_not be_nil
|
33
|
+
|
34
|
+
# Try a permanent override:
|
27
35
|
parser.root = :b
|
28
36
|
parser.parse('b').should_not be_nil
|
29
37
|
parser.parse('a').should be_nil
|
30
38
|
end
|
31
|
-
|
39
|
+
|
32
40
|
it "allows the requirement that all input be consumed to be disabled" do
|
33
41
|
parser.parse('ab').should be_nil
|
42
|
+
|
43
|
+
# Try a temporary override, and check it's not sticky:
|
44
|
+
result = parser.parse('ab', :consume_all_input => false)
|
45
|
+
result.should_not be_nil
|
46
|
+
result.interval.should == (0...1)
|
47
|
+
parser.parse('ab').should be_nil
|
48
|
+
|
49
|
+
# Now a permanent override:
|
34
50
|
parser.consume_all_input = false
|
35
51
|
result = parser.parse('ab')
|
36
52
|
result.should_not be_nil
|
37
53
|
result.interval.should == (0...1)
|
38
54
|
end
|
39
|
-
|
55
|
+
|
40
56
|
it "allows input to be parsed at a given index" do
|
41
57
|
parser.parse('ba').should be_nil
|
42
58
|
parser.parse('ba', :index => 1).should_not be_nil
|
59
|
+
# Check that the index defaults again to zero:
|
60
|
+
parser.parse('a').should_not be_nil
|
61
|
+
|
62
|
+
result = parser.parse('ba', :consume_all_input => false, :index => 1)
|
63
|
+
result.should_not be_nil
|
64
|
+
result.interval.should == (1...2)
|
43
65
|
end
|
44
|
-
|
66
|
+
|
45
67
|
end
|
46
68
|
|
47
69
|
describe Runtime::CompiledParser, "for a grammar with a choice between terminals" do
|
@@ -81,7 +103,7 @@ module CompiledParserSpec
|
|
81
103
|
before do
|
82
104
|
@parser = parser_class_under_test.new
|
83
105
|
end
|
84
|
-
|
106
|
+
|
85
107
|
it "is reset between parses" do
|
86
108
|
parser.parse('ac')
|
87
109
|
terminal_failures = parser.terminal_failures
|
metadata
CHANGED
@@ -1,12 +1,13 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: treetop
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
+
hash: 21
|
4
5
|
prerelease: false
|
5
6
|
segments:
|
6
7
|
- 1
|
7
8
|
- 4
|
8
|
-
-
|
9
|
-
version: 1.4.
|
9
|
+
- 9
|
10
|
+
version: 1.4.9
|
10
11
|
platform: ruby
|
11
12
|
authors:
|
12
13
|
- Nathan Sobo
|
@@ -14,7 +15,7 @@ autorequire: treetop
|
|
14
15
|
bindir: bin
|
15
16
|
cert_chain: []
|
16
17
|
|
17
|
-
date: 2010-
|
18
|
+
date: 2010-11-16 00:00:00 +11:00
|
18
19
|
default_executable:
|
19
20
|
dependencies:
|
20
21
|
- !ruby/object:Gem::Dependency
|
@@ -25,6 +26,7 @@ dependencies:
|
|
25
26
|
requirements:
|
26
27
|
- - ">="
|
27
28
|
- !ruby/object:Gem::Version
|
29
|
+
hash: 17
|
28
30
|
segments:
|
29
31
|
- 0
|
30
32
|
- 3
|
@@ -139,6 +141,19 @@ files:
|
|
139
141
|
- doc/index.markdown
|
140
142
|
- doc/pitfalls_and_advanced_techniques.markdown
|
141
143
|
- doc/semantic_interpretation.markdown
|
144
|
+
- doc/site/contribute.html
|
145
|
+
- doc/site/images/bottom_background.png
|
146
|
+
- doc/site/images/middle_background.png
|
147
|
+
- doc/site/images/paren_language_output.png
|
148
|
+
- doc/site/images/pivotal.gif
|
149
|
+
- doc/site/images/top_background.png
|
150
|
+
- doc/site/index.html
|
151
|
+
- doc/site/pitfalls_and_advanced_techniques.html
|
152
|
+
- doc/site/robots.txt
|
153
|
+
- doc/site/screen.css
|
154
|
+
- doc/site/semantic_interpretation.html
|
155
|
+
- doc/site/syntactic_recognition.html
|
156
|
+
- doc/site/using_in_ruby.html
|
142
157
|
- doc/site.rb
|
143
158
|
- doc/sitegen.rb
|
144
159
|
- doc/syntactic_recognition.markdown
|
@@ -167,6 +182,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
167
182
|
requirements:
|
168
183
|
- - ">="
|
169
184
|
- !ruby/object:Gem::Version
|
185
|
+
hash: 3
|
170
186
|
segments:
|
171
187
|
- 0
|
172
188
|
version: "0"
|
@@ -175,6 +191,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
175
191
|
requirements:
|
176
192
|
- - ">="
|
177
193
|
- !ruby/object:Gem::Version
|
194
|
+
hash: 3
|
178
195
|
segments:
|
179
196
|
- 0
|
180
197
|
version: "0"
|