antlr3 1.4.0 → 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/History.txt +59 -66
- data/java/RubyTarget.java +10 -5
- data/java/antlr-full-3.2.1.jar +0 -0
- data/lib/antlr3/debug.rb +8 -14
- data/lib/antlr3/debug/socket.rb +29 -29
- data/lib/antlr3/recognizers.rb +13 -23
- data/lib/antlr3/streams.rb +17 -8
- data/lib/antlr3/template.rb +11 -0
- data/lib/antlr3/template/group-lexer.rb +48 -48
- data/lib/antlr3/template/group-parser.rb +133 -94
- data/lib/antlr3/test/core-extensions.rb +1 -1
- data/lib/antlr3/token.rb +4 -4
- data/lib/antlr3/tree.rb +39 -35
- data/lib/antlr3/tree/debug.rb +3 -2
- data/lib/antlr3/version.rb +1 -1
- data/templates/Ruby.stg +27 -33
- data/templates/ST.stg +0 -3
- data/test/functional/ast-output/construction.rb +1 -1
- data/test/unit/test-exceptions.rb +12 -0
- metadata +4 -3
data/History.txt
CHANGED
@@ -1,66 +1,59 @@
|
|
1
|
-
|
2
|
-
|
3
|
-
|
4
|
-
|
5
|
-
|
6
|
-
|
7
|
-
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
*
|
44
|
-
|
45
|
-
|
46
|
-
|
47
|
-
|
48
|
-
|
49
|
-
|
50
|
-
|
51
|
-
|
52
|
-
|
53
|
-
|
54
|
-
|
55
|
-
|
56
|
-
|
57
|
-
|
58
|
-
|
59
|
-
|
60
|
-
* as handy as it is to add new functions to core classes, I moved most of the code in the core-extension directory to the new slightly-nebulous, catch-all dumping ground modules, ANTLR3::Util and ANTLR3::ClassMacros. String and Module methods defined in the extensions were only used in a small handful of places and thus it seemed intrusive to polute the core classes with these utility methods for little payoff.
|
61
|
-
* ANTLR classes now extend ANTLR3::ClassMacros when necessary. It defines 'abstract' to easily create abstract methods. It also provides a new method, #shared_attribute, which defines an attribute accessor with a writer
|
62
|
-
|
63
|
-
h1(update). V1.0.0 2009-10-14 Kyle Yetter <kcy5b@yahoo.com>
|
64
|
-
|
65
|
-
h2(major). Initial Release: Birthday!
|
66
|
-
|
1
|
+
=== 1.6.0 / 01-27-10
|
2
|
+
|
3
|
+
* 4 Minor Enhancements to runtime library
|
4
|
+
- CommonTokenStream and subclasses can now also accept another CommonTokenStream
|
5
|
+
object, serving sort of as a copy constructor. This provides a reasonable way
|
6
|
+
to cast a CommonTokenStream into a TokenRewriteStream and vice-versa
|
7
|
+
- CommonTreeNodeStream can take an options hash at the end of any of its various
|
8
|
+
signatures to provide values for attributes like token_stream
|
9
|
+
- If a parser is created with a CharacterStream object (i.e., StringStream), it
|
10
|
+
will also try to cast it by checking for an associated lexer (like it currently
|
11
|
+
does when created with String or IO objects).
|
12
|
+
- Template-output mode parsers have a new class method "load_templates( path )"
|
13
|
+
which will attempt to load the template group file at +path+ and, if successful,
|
14
|
+
set it to the parser's default template library. (See the CMinus example set for
|
15
|
+
an example of how this can be used)
|
16
|
+
|
17
|
+
* 2 bug fixes / 1 change to templates
|
18
|
+
- backtracking was broken in situations where recognition failed when
|
19
|
+
a semantic predicate was encountered as the template code did not
|
20
|
+
throw BacktrackingFailed when it was in backtracking mode instead of
|
21
|
+
PredicateFailed
|
22
|
+
- rule memoization also had a major bug in that rules that were memoized
|
23
|
+
after failure did not throw BacktrackingFailed when it was needed
|
24
|
+
- I changed `attribute scope' structures (not rule return scopes, but
|
25
|
+
the groups of attributes you can assign to rules with the `scope' keyword),
|
26
|
+
from being implemented as named constants in the class body to class
|
27
|
+
variables belonging to the recognizer. As class variables do not need
|
28
|
+
to follow naming conventions, this provides more freedom in naming without
|
29
|
+
all of the ugly name mangling. These are generally used privately by parsers,
|
30
|
+
so it shouldn't be too drastic of a change as to break end user's code.
|
31
|
+
|
32
|
+
=== 1.5.0 / 01-25-10
|
33
|
+
|
34
|
+
* 2 minor enhancements
|
35
|
+
- Added BaseTree#walk, which iterates in a top-down fashion across all notes in a tree.
|
36
|
+
- Added BaseTree#prune, which stops decent into the current node during BaseTree#walk.
|
37
|
+
|
38
|
+
* 1 optimization
|
39
|
+
- changed `closure loops' ( uses of the `*' or `+' modifiers ) from using
|
40
|
+
Kernel#loop to `while true ... end' style loops, which should be slightly
|
41
|
+
faster
|
42
|
+
|
43
|
+
* new test code
|
44
|
+
- test/functiona/template-output/template-output.rb serves as a basic
|
45
|
+
functional test implementation for the new template output mode
|
46
|
+
|
47
|
+
=== 1.4.0 / 01-17-10
|
48
|
+
|
49
|
+
* 1 major new feature
|
50
|
+
- finished basic implementation of a template output mode.
|
51
|
+
- lib/antlr3/template.rb defines a number of classes for
|
52
|
+
ERB-based template output translation
|
53
|
+
- templates/ST.stg defines the Ruby target templates to
|
54
|
+
generate template output mode recognizers
|
55
|
+
|
56
|
+
* new test code
|
57
|
+
- added template unit tests at test/unit/test-template.rb
|
58
|
+
- test/functional/template-output/template-output.rb serves as a basic
|
59
|
+
functional test implementation for the new template output mode
|
data/java/RubyTarget.java
CHANGED
@@ -262,6 +262,7 @@ extends Target
|
|
262
262
|
generator.getRecognizerFileName(grammar.name, grammar.type);
|
263
263
|
generator.write(outputFileST, fileName);
|
264
264
|
}
|
265
|
+
|
265
266
|
public String getTargetCharLiteralFromANTLRCharLiteral(
|
266
267
|
CodeGenerator generator,
|
267
268
|
String literal)
|
@@ -285,11 +286,13 @@ extends Target
|
|
285
286
|
|
286
287
|
return result;
|
287
288
|
}
|
289
|
+
|
288
290
|
public int getMaxCharValue(CodeGenerator generator)
|
289
291
|
{
|
290
|
-
//
|
292
|
+
// Versions before 1.9 do not support unicode
|
291
293
|
return 0xFF;
|
292
294
|
}
|
295
|
+
|
293
296
|
public String getTokenTypeAsTargetLabel(CodeGenerator generator, int ttype)
|
294
297
|
{
|
295
298
|
String name = generator.grammar.getTokenDisplayName(ttype);
|
@@ -299,6 +302,7 @@ extends Target
|
|
299
302
|
}
|
300
303
|
return name;
|
301
304
|
}
|
305
|
+
|
302
306
|
/** Is scope in @scope::name {action} valid for this kind of grammar?
|
303
307
|
* Targets like C++ may want to allow new scopes like headerfile or
|
304
308
|
* some such. The action names themselves are not policed at the
|
@@ -369,6 +373,7 @@ extends Target
|
|
369
373
|
}
|
370
374
|
return false;
|
371
375
|
}
|
376
|
+
|
372
377
|
/*
|
373
378
|
public String getTargetStringLiteralFromString(String s)
|
374
379
|
{
|
@@ -391,17 +396,17 @@ extends Target
|
|
391
396
|
System.out.print(result + "\n");
|
392
397
|
return result;
|
393
398
|
}
|
394
|
-
|
399
|
+
*/
|
400
|
+
|
395
401
|
public String encodeIntAsCharEscape(final int v) {
|
396
402
|
final int intValue;
|
397
403
|
|
398
|
-
if (v == 65535) {
|
404
|
+
if ( v == 65535 ) {
|
399
405
|
intValue = -1;
|
400
406
|
} else {
|
401
407
|
intValue = v;
|
402
408
|
}
|
403
409
|
|
404
|
-
return String.valueOf(intValue);
|
405
|
-
// return String.valueOf(v);
|
410
|
+
return String.valueOf( intValue );
|
406
411
|
}
|
407
412
|
}
|
data/java/antlr-full-3.2.1.jar
CHANGED
Binary file
|
data/lib/antlr3/debug.rb
CHANGED
@@ -228,21 +228,16 @@ module ParserEvents
|
|
228
228
|
@state.backtracking -= 1
|
229
229
|
end
|
230
230
|
|
231
|
-
def report_error(exc)
|
232
|
-
|
233
|
-
|
234
|
-
@debug_listener.recognition_exception(exc)
|
235
|
-
else
|
236
|
-
$stderr.puts(exc)
|
237
|
-
$stderr.puts(exc.backtrace)
|
238
|
-
end
|
231
|
+
def report_error( exc )
|
232
|
+
ANTLR3::RecognitionError === exc and
|
233
|
+
@debug_listener.recognition_exception( exc )
|
239
234
|
super
|
240
235
|
end
|
241
236
|
|
242
|
-
def missing_symbol(error, expected_type, follow)
|
237
|
+
def missing_symbol( error, expected_type, follow )
|
243
238
|
symbol = super
|
244
|
-
@debug_listener.consume_node(symbol)
|
245
|
-
return(symbol)
|
239
|
+
@debug_listener.consume_node( symbol )
|
240
|
+
return( symbol )
|
246
241
|
end
|
247
242
|
|
248
243
|
def in_rule(grammar_file, rule_name)
|
@@ -363,11 +358,10 @@ module TokenStream
|
|
363
358
|
return @last_marker
|
364
359
|
end
|
365
360
|
|
366
|
-
def rewind(marker = nil)
|
367
|
-
@debug_listener.rewind(marker)
|
361
|
+
def rewind( marker = nil, release = true )
|
362
|
+
@debug_listener.rewind( marker )
|
368
363
|
super
|
369
364
|
end
|
370
|
-
|
371
365
|
end
|
372
366
|
|
373
367
|
=begin rdoc ANTLR3::Debug::EventListener
|
data/lib/antlr3/debug/socket.rb
CHANGED
@@ -99,39 +99,39 @@ class EventSocketProxy
|
|
99
99
|
end
|
100
100
|
|
101
101
|
def enter_rule( grammar_file_name, rule_name )
|
102
|
-
transmit "%s\t%s\t%s",
|
102
|
+
transmit "%s\t%s\t%s", :enter_rule, grammar_file_name, rule_name
|
103
103
|
end
|
104
104
|
|
105
105
|
def enter_alternative( alt )
|
106
|
-
transmit "%s\t%s",
|
106
|
+
transmit "%s\t%s", :enter_alternative, alt
|
107
107
|
end
|
108
108
|
|
109
109
|
def exit_rule( grammar_file_name, rule_name )
|
110
|
-
transmit "%s\t%s\t%s",
|
110
|
+
transmit "%s\t%s\t%s", :exit_rule, grammar_file_name, rule_name
|
111
111
|
end
|
112
112
|
|
113
113
|
def enter_subrule( decision_number )
|
114
|
-
transmit "%s\t%i",
|
114
|
+
transmit "%s\t%i", :enter_subrule, decision_number
|
115
115
|
end
|
116
116
|
|
117
117
|
def exit_subrule( decision_number )
|
118
|
-
transmit "%s\t%i",
|
118
|
+
transmit "%s\t%i", :exit_subrule, decision_number
|
119
119
|
end
|
120
120
|
|
121
121
|
def enter_decision( decision_number )
|
122
|
-
transmit "%s\t%i",
|
122
|
+
transmit "%s\t%i", :enter_decision, decision_number
|
123
123
|
end
|
124
124
|
|
125
125
|
def exit_decision( decision_number )
|
126
|
-
transmit "%s\t%i",
|
126
|
+
transmit "%s\t%i", :exit_decision, decision_number
|
127
127
|
end
|
128
128
|
|
129
129
|
def consume_token( token )
|
130
|
-
transmit "%s\t%s",
|
130
|
+
transmit "%s\t%s", :consume_token, serialize_token( token )
|
131
131
|
end
|
132
132
|
|
133
133
|
def consume_hidden_token( token )
|
134
|
-
transmit "%s\t%s",
|
134
|
+
transmit "%s\t%s", :consume_hidden_token, serialize_token( token )
|
135
135
|
end
|
136
136
|
|
137
137
|
def look(i, item)
|
@@ -140,49 +140,49 @@ class EventSocketProxy
|
|
140
140
|
look_tree( i, item )
|
141
141
|
when nil
|
142
142
|
else
|
143
|
-
transmit "%s\t%i\t%s",
|
143
|
+
transmit "%s\t%i\t%s", :look, i, serialize_token(item)
|
144
144
|
end
|
145
145
|
end
|
146
146
|
|
147
147
|
def mark(i)
|
148
|
-
transmit "%s\t%i",
|
148
|
+
transmit "%s\t%i", :mark, i
|
149
149
|
end
|
150
150
|
|
151
151
|
def rewind(i = nil)
|
152
|
-
i ? transmit( "%s\t%i",
|
152
|
+
i ? transmit( "%s\t%i", :rewind, i ) : transmit( '%s', :rewind )
|
153
153
|
end
|
154
154
|
|
155
155
|
def begin_backtrack( level )
|
156
|
-
transmit "%s\t%i",
|
156
|
+
transmit "%s\t%i", :begin_backtrack, level
|
157
157
|
end
|
158
158
|
def end_backtrack( level, successful )
|
159
|
-
transmit "%s\t%i\t%p",
|
159
|
+
transmit "%s\t%i\t%p", :end_backtrack, level, (successful ? true : false)
|
160
160
|
end
|
161
161
|
|
162
162
|
def location( line, position )
|
163
|
-
transmit "%s\t%i\t%i",
|
163
|
+
transmit "%s\t%i\t%i", :location, line, position
|
164
164
|
end
|
165
165
|
|
166
166
|
def recognition_exception( exception )
|
167
|
-
transmit "%s\t%p\t%i\t%i\t%i",
|
167
|
+
transmit "%s\t%p\t%i\t%i\t%i", :recognition_exception, exception.class,
|
168
168
|
exception.index, exception.line, exception.column
|
169
169
|
end
|
170
170
|
|
171
171
|
def begin_resync
|
172
|
-
transmit '%s',
|
172
|
+
transmit '%s', :begin_resync
|
173
173
|
end
|
174
174
|
|
175
175
|
def end_resync
|
176
|
-
transmit '%s',
|
176
|
+
transmit '%s', :end_resync
|
177
177
|
end
|
178
178
|
|
179
179
|
def semantic_predicate( result, predicate )
|
180
180
|
pure_boolean = !(!result)
|
181
|
-
transmit "%s\t%s\t%s",
|
181
|
+
transmit "%s\t%s\t%s", :semantic_predicate, pure_boolean, escape_newlines( predicate )
|
182
182
|
end
|
183
183
|
|
184
184
|
def consume_node( tree )
|
185
|
-
transmit "%s\t%s",
|
185
|
+
transmit "%s\t%s", :consume_node, serialize_node( tree )
|
186
186
|
end
|
187
187
|
|
188
188
|
def adaptor
|
@@ -190,40 +190,40 @@ class EventSocketProxy
|
|
190
190
|
end
|
191
191
|
|
192
192
|
def look_tree( i, tree )
|
193
|
-
transmit "%s\t%s\t%s",
|
193
|
+
transmit "%s\t%s\t%s", :look_tree, i, serialize_node( tree )
|
194
194
|
end
|
195
195
|
|
196
196
|
def flat_node( tree )
|
197
|
-
transmit "%s\t%i",
|
197
|
+
transmit "%s\t%i", :flat_node, adaptor.unique_id( tree )
|
198
198
|
end
|
199
199
|
|
200
200
|
def error_node( tree )
|
201
|
-
transmit "%s\t%i\t%i\t%p",
|
201
|
+
transmit "%s\t%i\t%i\t%p", :error_node, adaptor.unique_id( tree ),
|
202
202
|
Token::INVALID_TOKEN_TYPE, escape_newlines( tree.to_s )
|
203
203
|
end
|
204
204
|
|
205
205
|
def create_node( node, token = nil )
|
206
206
|
if token
|
207
|
-
transmit "%s\t%i\t%i",
|
207
|
+
transmit "%s\t%i\t%i", :create_node, adaptor.unique_id( node ),
|
208
208
|
token.token_index
|
209
209
|
else
|
210
|
-
transmit "%s\t%i\t%i\t%p",
|
210
|
+
transmit "%s\t%i\t%i\t%p", :create_node, adaptor.unique_id( node ),
|
211
211
|
adaptor.type_of( node ), adaptor.text_of( node )
|
212
212
|
end
|
213
213
|
end
|
214
214
|
|
215
215
|
def become_root( new_root, old_root )
|
216
|
-
transmit "%s\t%i\t%i",
|
216
|
+
transmit "%s\t%i\t%i", :become_root, adaptor.unique_id( new_root ),
|
217
217
|
adaptor.unique_id( old_root )
|
218
218
|
end
|
219
219
|
|
220
220
|
def add_child( root, child )
|
221
|
-
transmit "%s\t%i\t%i",
|
221
|
+
transmit "%s\t%i\t%i", :add_child, adaptor.unique_id( root ),
|
222
222
|
adaptor.unique_id( child )
|
223
223
|
end
|
224
224
|
|
225
225
|
def set_token_boundaries( t, token_start_index, token_stop_index )
|
226
|
-
transmit "%s\t%i\t%i\t%i",
|
226
|
+
transmit "%s\t%i\t%i\t%i", :set_token_boundaries, adaptor.unique_id( t ),
|
227
227
|
token_start_index, token_stop_index
|
228
228
|
end
|
229
229
|
|
@@ -357,4 +357,4 @@ private
|
|
357
357
|
|
358
358
|
end # class RemoteEventSocketListener
|
359
359
|
end # module Debug
|
360
|
-
end # module ANTLR3
|
360
|
+
end # module ANTLR3
|
data/lib/antlr3/recognizers.rb
CHANGED
@@ -361,8 +361,9 @@ class BaseRecognizer
|
|
361
361
|
# to display the error info on $stderr.
|
362
362
|
def report_error(e = $!)
|
363
363
|
@state.error_recovery and return
|
364
|
+
@state.syntax_errors += 1
|
364
365
|
@state.error_recovery = true
|
365
|
-
display_recognition_error(e)
|
366
|
+
display_recognition_error( e )
|
366
367
|
end
|
367
368
|
|
368
369
|
# error reporting hook for presenting the information
|
@@ -789,18 +790,19 @@ class BaseRecognizer
|
|
789
790
|
alias backtracking backtracking_level
|
790
791
|
alias backtracking= backtracking_level=
|
791
792
|
|
792
|
-
def rule_memoization(rule, start_index)
|
793
|
-
@state.rule_memory[rule] ||= Hash.new(MEMO_RULE_UNKNOWN)
|
794
|
-
@state.rule_memory[rule][start_index]
|
793
|
+
def rule_memoization( rule, start_index )
|
794
|
+
@state.rule_memory[ rule ] ||= Hash.new( MEMO_RULE_UNKNOWN )
|
795
|
+
@state.rule_memory[ rule ][ start_index ]
|
795
796
|
end
|
796
797
|
|
797
|
-
def already_parsed_rule?(rule)
|
798
|
-
stop_index = rule_memoization(rule, @input.index)
|
798
|
+
def already_parsed_rule?( rule )
|
799
|
+
stop_index = rule_memoization( rule, @input.index )
|
799
800
|
case stop_index
|
800
801
|
when MEMO_RULE_UNKNOWN then return false
|
801
|
-
when MEMO_RULE_FAILED
|
802
|
+
when MEMO_RULE_FAILED
|
803
|
+
raise BacktrackingFailed
|
802
804
|
else
|
803
|
-
@input.seek(stop_index + 1)
|
805
|
+
@input.seek( stop_index + 1 )
|
804
806
|
end
|
805
807
|
return true
|
806
808
|
end
|
@@ -923,18 +925,6 @@ class Lexer < BaseRecognizer
|
|
923
925
|
block_given? ? yield(main) : main.execute(argv)
|
924
926
|
end
|
925
927
|
|
926
|
-
def self.associated_parser
|
927
|
-
@grammar_home and @grammar_home::Parser
|
928
|
-
rescue NameError
|
929
|
-
grammar_name = @grammar_home.name.split("::").last
|
930
|
-
begin
|
931
|
-
require "#{grammar_name}Parser"
|
932
|
-
rescue LoadError => e
|
933
|
-
return nil
|
934
|
-
end
|
935
|
-
return @grammar_home::Parser rescue nil
|
936
|
-
end
|
937
|
-
|
938
928
|
def self.associated_parser
|
939
929
|
@associated_parser ||= begin
|
940
930
|
@grammar_home and @grammar_home::Parser
|
@@ -949,7 +939,7 @@ class Lexer < BaseRecognizer
|
|
949
939
|
end
|
950
940
|
|
951
941
|
def initialize(input, options = {})
|
952
|
-
super(options)
|
942
|
+
super( options )
|
953
943
|
@input =
|
954
944
|
case input
|
955
945
|
when ::String then StringStream.new(input, options)
|
@@ -1009,7 +999,7 @@ class Lexer < BaseRecognizer
|
|
1009
999
|
@input.source_name
|
1010
1000
|
end
|
1011
1001
|
|
1012
|
-
def emit(token =
|
1002
|
+
def emit( token = @state.token )
|
1013
1003
|
token ||= create_token
|
1014
1004
|
@state.token = token
|
1015
1005
|
return token
|
@@ -1254,7 +1244,7 @@ private
|
|
1254
1244
|
def cast_input( input, options )
|
1255
1245
|
case input
|
1256
1246
|
when TokenSource then CommonTokenStream.new( input, options )
|
1257
|
-
when IO, String
|
1247
|
+
when IO, String, CharacterStream
|
1258
1248
|
if lexer_class = self.class.associated_lexer
|
1259
1249
|
CommonTokenStream.new( lexer_class.new( input, options ), options )
|
1260
1250
|
else
|