treetop 1.4.8 → 1.4.9

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  #Using Treetop Grammars in Ruby
2
2
  ##Using the Command Line Compiler
3
- You can `.treetop` files into Ruby source code with the `tt` command line script. `tt` takes an list of files with a `.treetop` extension and compiles them into `.rb` files of the same name. You can then `require` these files like any other Ruby script. Alternately, you can supply just one `.treetop` file and a `-o` flag to name specify the name of the output file. Improvements to this compilation script are welcome.
3
+ You can compile `.treetop` files into Ruby source code with the `tt` command line script. `tt` takes an list of files with a `.treetop` extension and compiles them into `.rb` files of the same name. You can then `require` these files like any other Ruby script. Alternately, you can supply just one `.treetop` file and a `-o` flag to name specify the name of the output file. Improvements to this compilation script are welcome.
4
4
 
5
5
  tt foo.treetop bar.treetop
6
6
  tt foo.treetop -o foogrammar.rb
@@ -16,7 +16,10 @@ In order to use Polyglot dynamic loading of `.treetop` or `.tt` files though, yo
16
16
  in order to use Polyglot auto loading with Treetop in Ruby.
17
17
 
18
18
  ##Instantiating and Using Parsers
19
- If a grammar by the name of `Foo` is defined, the compiled Ruby source will define a `FooParser` class. To parse input, create an instance and call its `parse` method with a string. The parser will return the syntax tree of the match or `nil` if there is a failure.
19
+ If a grammar by the name of `Foo` is defined, the compiled Ruby source will define a `FooParser` class.
20
+ To parse input, create an instance and call its `parse` method with a string.
21
+ The parser will return the syntax tree of the match or `nil` if there is a failure.
22
+ Note that by default, the parser will fail unless *all* input is consumed.
20
23
 
21
24
  Treetop.load "arithmetic"
22
25
 
@@ -26,3 +29,77 @@ If a grammar by the name of `Foo` is defined, the compiled Ruby source will defi
26
29
  else
27
30
  puts 'failure'
28
31
  end
32
+
33
+ ##Parser Options
34
+ A Treetop parser has several options you may set.
35
+ Some are settable permanently by methods on the parser, but all may be passed in as options to the `parse` method.
36
+
37
+ parser = ArithmeticParser.new
38
+ input = 'x = 2; y = x+3;'
39
+
40
+ # Temporarily override an option:
41
+ result1 = parser.parse(input, :consume_all_input => false)
42
+ puts "consumed #{parser.index} characters"
43
+
44
+ parser.consume_all_input = false
45
+ result1 = parser.parse(input)
46
+ puts "consumed #{parser.index} characters"
47
+
48
+ # Continue the parse with the next character:
49
+ result2 = parser.parse(input, :index => parser.index)
50
+
51
+ # Parse, but match rule `variable` instead of the normal root rule:
52
+ parser.parse(input, :root => :variable)
53
+ parser.root = :variable # Permanent setting
54
+
55
+ If you have a statement-oriented language, you can save memory by parsing just one statement at a time,
56
+ and discarding the parse tree after each statement.
57
+
58
+
59
+ ##Learning From Failure
60
+ If a parse fails, it returns nil. In this case, you can ask the parser for an explanation.
61
+ The failure reasons include the terminal nodes which were tried at the furthermost point the parse reached.
62
+
63
+ parser = ArithmeticParser.new
64
+ result = parser.parse('4+=3')
65
+
66
+ if !result
67
+ puts parser.failure_reason
68
+ puts parser.failure_line
69
+ puts parser.failure_column
70
+ end
71
+
72
+ =>
73
+ Expected one of (, - at line 1, column 3 (byte 3) after +
74
+ 1
75
+ 3
76
+
77
+
78
+ ##Using Parse Results
79
+ Please don't try to walk down the syntax tree yourself, and please don't use the tree as your own convenient data structure.
80
+ It contains many more nodes than your application needs, often even more than one for every character of input.
81
+
82
+ parser = ArithmeticParser.new
83
+ p parser.parse('2+3')
84
+
85
+ =>
86
+ SyntaxNode+Additive1 offset=0, "2+3" (multitive):
87
+ SyntaxNode+Multitive1 offset=0, "2" (primary):
88
+ SyntaxNode+Number0 offset=0, "2":
89
+ SyntaxNode offset=0, ""
90
+ SyntaxNode offset=0, "2"
91
+ SyntaxNode offset=1, ""
92
+ SyntaxNode offset=1, ""
93
+ SyntaxNode offset=1, "+3":
94
+ SyntaxNode+Additive0 offset=1, "+3" (multitive):
95
+ SyntaxNode offset=1, "+"
96
+ SyntaxNode+Multitive1 offset=2, "3" (primary):
97
+ SyntaxNode+Number0 offset=2, "3":
98
+ SyntaxNode offset=2, ""
99
+ SyntaxNode offset=2, "3"
100
+ SyntaxNode offset=3, ""
101
+ SyntaxNode offset=3, ""
102
+
103
+ Instead, add methods to the root rule which return the information you require in a sensible form.
104
+ Each rule can call its sub-rules, and this method of walking the syntax tree is much preferable to
105
+ attempting to walk it from the outside.
@@ -4,7 +4,10 @@ grammar Arithmetic
4
4
  end
5
5
 
6
6
  rule comparative
7
- operand_1:additive space operator:equality_op space operand_2:additive <BinaryOperation>
7
+ head:additive
8
+ tail:(
9
+ space operator:equality_op
10
+ space operand:additive)* <BinaryOperation>
8
11
  end
9
12
 
10
13
  rule equality_op
@@ -16,11 +19,10 @@ grammar Arithmetic
16
19
  end
17
20
 
18
21
  rule additive
19
- operand_1:multitive
20
- space operator:additive_op space
21
- operand_2:additive <BinaryOperation>
22
- /
23
- multitive
22
+ head:multitive
23
+ tail:(
24
+ space operator:additive_op
25
+ space operand:multitive)* <BinaryOperation>
24
26
  end
25
27
 
26
28
  rule additive_op
@@ -38,11 +40,10 @@ grammar Arithmetic
38
40
  end
39
41
 
40
42
  rule multitive
41
- operand_1:primary
42
- space operator:multitive_op space
43
- operand_2:multitive <BinaryOperation>
44
- /
45
- primary
43
+ head:primary
44
+ tail:(
45
+ space operator:multitive_op
46
+ space operand:primary)* <BinaryOperation>
46
47
  end
47
48
 
48
49
  rule multitive_op
@@ -94,4 +95,4 @@ grammar Arithmetic
94
95
  rule space
95
96
  ' '*
96
97
  end
97
- end
98
+ end
@@ -1,7 +1,9 @@
1
1
  module Arithmetic
2
2
  class BinaryOperation < Treetop::Runtime::SyntaxNode
3
3
  def eval(env={})
4
- operator.apply(operand_1.eval(env), operand_2.eval(env))
4
+ tail.elements.inject(head.eval(env)) do |value, element|
5
+ element.operator.apply(value, element.operand.eval(env))
6
+ end
5
7
  end
6
8
  end
7
- end
9
+ end
@@ -43,6 +43,10 @@ class ArithmeticParserTest < Test::Unit::TestCase
43
43
  assert_equal 11, parse('1 + 2 * 3 + 4').eval
44
44
  end
45
45
 
46
+ def test_left_to_right
47
+ assert_equal 2, parse('5 - 2 - 1').eval
48
+ end
49
+
46
50
  def test_parentheses
47
51
  assert_equal 25, parse('(5 + x) * (10 - y)').eval('x' => 0, 'y' => 5)
48
52
  end
@@ -15,8 +15,9 @@ module Treetop
15
15
  def parse(input, options = {})
16
16
  prepare_to_parse(input)
17
17
  @index = options[:index] if options[:index]
18
- result = send("_nt_#{root}")
19
- return nil if (consume_all_input? && index != input.size)
18
+ result = send("_nt_#{options[:root] || root}")
19
+ should_consume_all = options.include?(:consume_all_input) ? options[:consume_all_input] : consume_all_input?
20
+ return nil if (should_consume_all && index != input.size)
20
21
  return SyntaxNode.new(input, index...(index + 1)) if result == true
21
22
  return result
22
23
  end
@@ -2,7 +2,7 @@ module Treetop #:nodoc:
2
2
  module VERSION #:nodoc:
3
3
  MAJOR = 1
4
4
  MINOR = 4
5
- TINY = 8
5
+ TINY = 9
6
6
 
7
7
  STRING = [MAJOR, MINOR, TINY].join('.')
8
8
  end
@@ -3,45 +3,67 @@ require 'spec_helper'
3
3
  module CompiledParserSpec
4
4
  describe Runtime::CompiledParser, "for a grammar with two rules" do
5
5
  attr_reader :parser
6
-
6
+
7
7
  testing_grammar %{
8
8
  grammar TwoRules
9
9
  rule a
10
10
  'a'
11
11
  end
12
-
12
+
13
13
  rule b
14
14
  'b'
15
15
  end
16
16
  end
17
17
  }
18
-
18
+
19
19
  before do
20
20
  @parser = parser_class_under_test.new
21
21
  end
22
-
22
+
23
23
  it "allows its root to be specified" do
24
24
  parser.parse('a').should_not be_nil
25
25
  parser.parse('b').should be_nil
26
-
26
+
27
+ # Check that the temporary-override works:
28
+ parser.parse('b', :root => :b).should_not be_nil
29
+ parser.parse('a', :root => :b).should be_nil
30
+
31
+ # Check that the temporary-override isn't sticky:
32
+ parser.parse('a').should_not be_nil
33
+
34
+ # Try a permanent override:
27
35
  parser.root = :b
28
36
  parser.parse('b').should_not be_nil
29
37
  parser.parse('a').should be_nil
30
38
  end
31
-
39
+
32
40
  it "allows the requirement that all input be consumed to be disabled" do
33
41
  parser.parse('ab').should be_nil
42
+
43
+ # Try a temporary override, and check it's not sticky:
44
+ result = parser.parse('ab', :consume_all_input => false)
45
+ result.should_not be_nil
46
+ result.interval.should == (0...1)
47
+ parser.parse('ab').should be_nil
48
+
49
+ # Now a permanent override:
34
50
  parser.consume_all_input = false
35
51
  result = parser.parse('ab')
36
52
  result.should_not be_nil
37
53
  result.interval.should == (0...1)
38
54
  end
39
-
55
+
40
56
  it "allows input to be parsed at a given index" do
41
57
  parser.parse('ba').should be_nil
42
58
  parser.parse('ba', :index => 1).should_not be_nil
59
+ # Check that the index defaults again to zero:
60
+ parser.parse('a').should_not be_nil
61
+
62
+ result = parser.parse('ba', :consume_all_input => false, :index => 1)
63
+ result.should_not be_nil
64
+ result.interval.should == (1...2)
43
65
  end
44
-
66
+
45
67
  end
46
68
 
47
69
  describe Runtime::CompiledParser, "for a grammar with a choice between terminals" do
@@ -81,7 +103,7 @@ module CompiledParserSpec
81
103
  before do
82
104
  @parser = parser_class_under_test.new
83
105
  end
84
-
106
+
85
107
  it "is reset between parses" do
86
108
  parser.parse('ac')
87
109
  terminal_failures = parser.terminal_failures
metadata CHANGED
@@ -1,12 +1,13 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: treetop
3
3
  version: !ruby/object:Gem::Version
4
+ hash: 21
4
5
  prerelease: false
5
6
  segments:
6
7
  - 1
7
8
  - 4
8
- - 8
9
- version: 1.4.8
9
+ - 9
10
+ version: 1.4.9
10
11
  platform: ruby
11
12
  authors:
12
13
  - Nathan Sobo
@@ -14,7 +15,7 @@ autorequire: treetop
14
15
  bindir: bin
15
16
  cert_chain: []
16
17
 
17
- date: 2010-05-31 00:00:00 +10:00
18
+ date: 2010-11-16 00:00:00 +11:00
18
19
  default_executable:
19
20
  dependencies:
20
21
  - !ruby/object:Gem::Dependency
@@ -25,6 +26,7 @@ dependencies:
25
26
  requirements:
26
27
  - - ">="
27
28
  - !ruby/object:Gem::Version
29
+ hash: 17
28
30
  segments:
29
31
  - 0
30
32
  - 3
@@ -139,6 +141,19 @@ files:
139
141
  - doc/index.markdown
140
142
  - doc/pitfalls_and_advanced_techniques.markdown
141
143
  - doc/semantic_interpretation.markdown
144
+ - doc/site/contribute.html
145
+ - doc/site/images/bottom_background.png
146
+ - doc/site/images/middle_background.png
147
+ - doc/site/images/paren_language_output.png
148
+ - doc/site/images/pivotal.gif
149
+ - doc/site/images/top_background.png
150
+ - doc/site/index.html
151
+ - doc/site/pitfalls_and_advanced_techniques.html
152
+ - doc/site/robots.txt
153
+ - doc/site/screen.css
154
+ - doc/site/semantic_interpretation.html
155
+ - doc/site/syntactic_recognition.html
156
+ - doc/site/using_in_ruby.html
142
157
  - doc/site.rb
143
158
  - doc/sitegen.rb
144
159
  - doc/syntactic_recognition.markdown
@@ -167,6 +182,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
167
182
  requirements:
168
183
  - - ">="
169
184
  - !ruby/object:Gem::Version
185
+ hash: 3
170
186
  segments:
171
187
  - 0
172
188
  version: "0"
@@ -175,6 +191,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
175
191
  requirements:
176
192
  - - ">="
177
193
  - !ruby/object:Gem::Version
194
+ hash: 3
178
195
  segments:
179
196
  - 0
180
197
  version: "0"