antlr3 1.4.0 → 1.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/History.txt CHANGED
@@ -1,66 +1,59 @@
1
- h1(update). V1.1.1 2009-10-24 Kyle Yetter <kcy5b@yahoo.com>
2
-
3
- h2(minor). minor alteration to TokenScheme
4
-
5
- * TokenScheme#register_name would raise an error if the token type was anonymous (i.e. T__19 or such) and the new name didn't have the format "'a string'". That works well for ANTLR tokens, but it's generally pointless, so I got rid of those restrictions.
6
-
7
- |\2. MODIFIED |
8
- | lib/antlr3/token.rb | made minor changes to ANTLR3::TokenScheme#register_name |
9
-
10
- h1(update). V1.1.0 2009-10-19 Kyle Yetter <kcy5b@yahoo.com>
11
-
12
- h2(major). moved FilterMode
13
-
14
- * since <tt>ANTLR3::Lexer::FilterMode</tt> is only necessary in lexers that specify filter=true, I moved it out of antlr3/recognizers.rb to its own file, antlr3/modes/filter.rb.
15
-
16
- * the FilterMode module is no longer in the Lexer namespace, but in the broader ANTLR3 namespace ... i.e. module ANTLR3::Lexer::FilterMode ==> module ANTLR3::FilterMode
17
-
18
- * I added an autoload entry to the ANTLR3 module in antlr3.rb for FilterMode, thus the module is autoloaded as needed whenever the constant ANTLR3::FilterMode is referenced
19
-
20
- |\2. MODIFIED |
21
- | lib/antlr3/recognizers.rb | removed Lexer::FilterMode definition |
22
- | lib/antlr3.rb | added autoload :FilterMode entry |
23
- |\2. CREATED |
24
- | lib/antlr3/modes/filter.rb | contains FilterMode module extracted from recognizers.rb |
25
-
26
- h2(major). new module ASTBuilder
27
-
28
- * created a new module, ANTLR3::ASTBuilder
29
- * all tree-building recognizers (parsers and tree parsers with the output=AST option) include this module
30
- * provides a way to test a recognizer to see if it is an AST-building recognizer (rec.is_a?(ANTLR3::ASTBuilder)) and it also added a number of methods to help clean up generated AST-parser code
31
-
32
- |\2. MODIFIED |
33
- | lib/antlr3.rb | added the usual autoload entry for ASTBuilder |
34
- | templates/AST.stg | added ``include ANTLR3::ASTBuilder'' to automatically include the module on all AST-output recognizers |
35
- |\2. CREATED |
36
- | lib/antlr3/modes/ast-builder.rb | defined new module ANTLR3::ASTBuilder |
37
-
38
- h2(major). refactored debug and profile modes
39
-
40
- * began process of organizing/cleaning-up the ANTLR3::Debug module (in file lib/antlr3/debug.rb), which contains all runtime support code and features for --debug and --profile option code output.
41
- * extracted profiling-oriented classes and modules from the Debug module and moved them to a new file, lib/antlr3/profile.rb.
42
- * since usually you're only using one particular type of Debug::EventListener at a time, I moved the six or so different EventListener classes out of Debug and into their own respective files in the antlr3/debug directory
43
- * since tree-oriented debugging hooks (Debug::TreeAdaptor and such) are only necessary for tree-oriented code, and thus are not part of the essential core, I moved the tree-related event modules out of Debug and into a separate file, lib/antlr3/tree/debug.rb
44
- * debug.rb now only defines the core EventListener interface module and the event-hook modules ParserEvents, TokenStream, and the expanded RecognizerSharedState class
45
- * as with most classes that aren't loaded by default in the runtime library, I strategically placed autoload statements in appropriate modules to autoload any of the Debug modules and classes I yanked out of the debug.rb, so there shouldn't be any need for extra 'require' statements to use the classes
46
-
47
- |\2. MODIFIED |
48
- | lib/antlr3/debug.rb | removed definitions: EventSocketProxy, RemoteEventSocketListener, TraceEventListener, RuleTracer, EventHub, TreeNodeStream, TreeAdaptor, ParserProfilingEvents, Profiler, Profiler::Profile, Profiler::DataSet |
49
- |\2. CREATED |
50
- | lib/antlr3/debug/event-hub.rb | new home for Debug::EventHub |
51
- | lib/antlr3/debug/record-event-listener.rb | new home for Debug::RecordEventListener |
52
- | lib/antlr3/debug/rule-tracer.rb | new home for Debug::RuleTracer |
53
- | lib/antlr3/debug/socket.rb | new home for Debug::EventSocketProxy and Debug::RemoteEventSocketListener |
54
- | lib/antlr3/debug/trace-event-listener.rb | new home for Debug::TraceEventListener |
55
- | lib/antlr3/profile.rb | new profile-oriented module, Profile, contains former debug classes Profile, ProfileEvents, and Profiler |
56
- | lib/antlr3/tree/debug.rb | new home for Debug::TreeAdaptor and Debug::TreeNodeStream |
57
-
58
- h2(minor). moved most of the core-extension code to new Util module
59
-
60
- * as handy as it is to add new functions to core classes, I moved most of the code in the core-extension directory to the new slightly-nebulous, catch-all dumping ground modules, ANTLR3::Util and ANTLR3::ClassMacros. String and Module methods defined in the extensions were only used in a small handful of places and thus it seemed intrusive to polute the core classes with these utility methods for little payoff.
61
- * ANTLR classes now extend ANTLR3::ClassMacros when necessary. It defines 'abstract' to easily create abstract methods. It also provides a new method, #shared_attribute, which defines an attribute accessor with a writer
62
-
63
- h1(update). V1.0.0 2009-10-14 Kyle Yetter <kcy5b@yahoo.com>
64
-
65
- h2(major). Initial Release: Birthday!
66
-
1
+ === 1.6.0 / 01-27-10
2
+
3
+ * 4 Minor Enhancements to runtime library
4
+ - CommonTokenStream and subclasses can now also accept another CommonTokenStream
5
+ object, serving sort of as a copy constructor. This provides a reasonable way
6
+ to cast a CommonTokenStream into a TokenRewriteStream and vice-versa
7
+ - CommonTreeNodeStream can take an options hash at the end of any of its various
8
+ signatures to provide values for attributes like token_stream
9
+ - If a parser is created with a CharacterStream object (i.e., StringStream), it
10
+ will also try to cast it by checking for an associated lexer (like it currently
11
+ does when created with String or IO objects).
12
+ - Template-output mode parsers have a new class method "load_templates( path )"
13
+ which will attempt to load the template group file at +path+ and, if successful,
14
+ set it to the parser's default template library. (See the CMinus example set for
15
+ an example of how this can be used)
16
+
17
+ * 2 bug fixes / 1 change to templates
18
+ - backtracking was broken in situations where recognition failed when
19
+ a semantic predicate was encountered as the template code did not
20
+ throw BacktrackingFailed when it was in backtracking mode instead of
21
+ PredicateFailed
22
+ - rule memoization also had a major bug in that rules that were memoized
23
+ after failure did not throw BacktrackingFailed when it was needed
24
+ - I changed `attribute scope' structures (not rule return scopes, but
25
+ the groups of attributes you can assign to rules with the `scope' keyword),
26
+ from being implemented as named constants in the class body to class
27
+ variables belonging to the recognizer. As class variables do not need
28
+ to follow naming conventions, this provides more freedom in naming without
29
+ all of the ugly name mangling. These are generally used privately by parsers,
30
+ so it shouldn't be too drastic of a change as to break end user's code.
31
+
32
+ === 1.5.0 / 01-25-10
33
+
34
+ * 2 minor enhancements
35
+ - Added BaseTree#walk, which iterates in a top-down fashion across all notes in a tree.
36
+ - Added BaseTree#prune, which stops decent into the current node during BaseTree#walk.
37
+
38
+ * 1 optimization
39
+ - changed `closure loops' ( uses of the `*' or `+' modifiers ) from using
40
+ Kernel#loop to `while true ... end' style loops, which should be slightly
41
+ faster
42
+
43
+ * new test code
44
+ - test/functiona/template-output/template-output.rb serves as a basic
45
+ functional test implementation for the new template output mode
46
+
47
+ === 1.4.0 / 01-17-10
48
+
49
+ * 1 major new feature
50
+ - finished basic implementation of a template output mode.
51
+ - lib/antlr3/template.rb defines a number of classes for
52
+ ERB-based template output translation
53
+ - templates/ST.stg defines the Ruby target templates to
54
+ generate template output mode recognizers
55
+
56
+ * new test code
57
+ - added template unit tests at test/unit/test-template.rb
58
+ - test/functional/template-output/template-output.rb serves as a basic
59
+ functional test implementation for the new template output mode
data/java/RubyTarget.java CHANGED
@@ -262,6 +262,7 @@ extends Target
262
262
  generator.getRecognizerFileName(grammar.name, grammar.type);
263
263
  generator.write(outputFileST, fileName);
264
264
  }
265
+
265
266
  public String getTargetCharLiteralFromANTLRCharLiteral(
266
267
  CodeGenerator generator,
267
268
  String literal)
@@ -285,11 +286,13 @@ extends Target
285
286
 
286
287
  return result;
287
288
  }
289
+
288
290
  public int getMaxCharValue(CodeGenerator generator)
289
291
  {
290
- // we don't support unicode, yet.
292
+ // Versions before 1.9 do not support unicode
291
293
  return 0xFF;
292
294
  }
295
+
293
296
  public String getTokenTypeAsTargetLabel(CodeGenerator generator, int ttype)
294
297
  {
295
298
  String name = generator.grammar.getTokenDisplayName(ttype);
@@ -299,6 +302,7 @@ extends Target
299
302
  }
300
303
  return name;
301
304
  }
305
+
302
306
  /** Is scope in @scope::name {action} valid for this kind of grammar?
303
307
  * Targets like C++ may want to allow new scopes like headerfile or
304
308
  * some such. The action names themselves are not policed at the
@@ -369,6 +373,7 @@ extends Target
369
373
  }
370
374
  return false;
371
375
  }
376
+
372
377
  /*
373
378
  public String getTargetStringLiteralFromString(String s)
374
379
  {
@@ -391,17 +396,17 @@ extends Target
391
396
  System.out.print(result + "\n");
392
397
  return result;
393
398
  }
394
- */
399
+ */
400
+
395
401
  public String encodeIntAsCharEscape(final int v) {
396
402
  final int intValue;
397
403
 
398
- if (v == 65535) {
404
+ if ( v == 65535 ) {
399
405
  intValue = -1;
400
406
  } else {
401
407
  intValue = v;
402
408
  }
403
409
 
404
- return String.valueOf(intValue);
405
- // return String.valueOf(v);
410
+ return String.valueOf( intValue );
406
411
  }
407
412
  }
Binary file
data/lib/antlr3/debug.rb CHANGED
@@ -228,21 +228,16 @@ module ParserEvents
228
228
  @state.backtracking -= 1
229
229
  end
230
230
 
231
- def report_error(exc)
232
- case exc
233
- when ANTLR3::Error::RecognitionError
234
- @debug_listener.recognition_exception(exc)
235
- else
236
- $stderr.puts(exc)
237
- $stderr.puts(exc.backtrace)
238
- end
231
+ def report_error( exc )
232
+ ANTLR3::RecognitionError === exc and
233
+ @debug_listener.recognition_exception( exc )
239
234
  super
240
235
  end
241
236
 
242
- def missing_symbol(error, expected_type, follow)
237
+ def missing_symbol( error, expected_type, follow )
243
238
  symbol = super
244
- @debug_listener.consume_node(symbol)
245
- return(symbol)
239
+ @debug_listener.consume_node( symbol )
240
+ return( symbol )
246
241
  end
247
242
 
248
243
  def in_rule(grammar_file, rule_name)
@@ -363,11 +358,10 @@ module TokenStream
363
358
  return @last_marker
364
359
  end
365
360
 
366
- def rewind(marker = nil)
367
- @debug_listener.rewind(marker)
361
+ def rewind( marker = nil, release = true )
362
+ @debug_listener.rewind( marker )
368
363
  super
369
364
  end
370
-
371
365
  end
372
366
 
373
367
  =begin rdoc ANTLR3::Debug::EventListener
@@ -99,39 +99,39 @@ class EventSocketProxy
99
99
  end
100
100
 
101
101
  def enter_rule( grammar_file_name, rule_name )
102
- transmit "%s\t%s\t%s", __method__, grammar_file_name, rule_name
102
+ transmit "%s\t%s\t%s", :enter_rule, grammar_file_name, rule_name
103
103
  end
104
104
 
105
105
  def enter_alternative( alt )
106
- transmit "%s\t%s", __method__, alt
106
+ transmit "%s\t%s", :enter_alternative, alt
107
107
  end
108
108
 
109
109
  def exit_rule( grammar_file_name, rule_name )
110
- transmit "%s\t%s\t%s", __method__, grammar_file_name, rule_name
110
+ transmit "%s\t%s\t%s", :exit_rule, grammar_file_name, rule_name
111
111
  end
112
112
 
113
113
  def enter_subrule( decision_number )
114
- transmit "%s\t%i", __method__, decision_number
114
+ transmit "%s\t%i", :enter_subrule, decision_number
115
115
  end
116
116
 
117
117
  def exit_subrule( decision_number )
118
- transmit "%s\t%i", __method__, decision_number
118
+ transmit "%s\t%i", :exit_subrule, decision_number
119
119
  end
120
120
 
121
121
  def enter_decision( decision_number )
122
- transmit "%s\t%i", __method__, decision_number
122
+ transmit "%s\t%i", :enter_decision, decision_number
123
123
  end
124
124
 
125
125
  def exit_decision( decision_number )
126
- transmit "%s\t%i", __method__, decision_number
126
+ transmit "%s\t%i", :exit_decision, decision_number
127
127
  end
128
128
 
129
129
  def consume_token( token )
130
- transmit "%s\t%s", __method__, serialize_token( token )
130
+ transmit "%s\t%s", :consume_token, serialize_token( token )
131
131
  end
132
132
 
133
133
  def consume_hidden_token( token )
134
- transmit "%s\t%s", __method__, serialize_token( token )
134
+ transmit "%s\t%s", :consume_hidden_token, serialize_token( token )
135
135
  end
136
136
 
137
137
  def look(i, item)
@@ -140,49 +140,49 @@ class EventSocketProxy
140
140
  look_tree( i, item )
141
141
  when nil
142
142
  else
143
- transmit "%s\t%i\t%s", __method__, i, serialize_token(item)
143
+ transmit "%s\t%i\t%s", :look, i, serialize_token(item)
144
144
  end
145
145
  end
146
146
 
147
147
  def mark(i)
148
- transmit "%s\t%i", __method__, i
148
+ transmit "%s\t%i", :mark, i
149
149
  end
150
150
 
151
151
  def rewind(i = nil)
152
- i ? transmit( "%s\t%i", __method__, i ) : transmit( '%s', __method__ )
152
+ i ? transmit( "%s\t%i", :rewind, i ) : transmit( '%s', :rewind )
153
153
  end
154
154
 
155
155
  def begin_backtrack( level )
156
- transmit "%s\t%i", __method__, level
156
+ transmit "%s\t%i", :begin_backtrack, level
157
157
  end
158
158
  def end_backtrack( level, successful )
159
- transmit "%s\t%i\t%p", __method__, level, (successful ? true : false)
159
+ transmit "%s\t%i\t%p", :end_backtrack, level, (successful ? true : false)
160
160
  end
161
161
 
162
162
  def location( line, position )
163
- transmit "%s\t%i\t%i", __method__, line, position
163
+ transmit "%s\t%i\t%i", :location, line, position
164
164
  end
165
165
 
166
166
  def recognition_exception( exception )
167
- transmit "%s\t%p\t%i\t%i\t%i", __method__, exception.class,
167
+ transmit "%s\t%p\t%i\t%i\t%i", :recognition_exception, exception.class,
168
168
  exception.index, exception.line, exception.column
169
169
  end
170
170
 
171
171
  def begin_resync
172
- transmit '%s', __method__
172
+ transmit '%s', :begin_resync
173
173
  end
174
174
 
175
175
  def end_resync
176
- transmit '%s', __method__
176
+ transmit '%s', :end_resync
177
177
  end
178
178
 
179
179
  def semantic_predicate( result, predicate )
180
180
  pure_boolean = !(!result)
181
- transmit "%s\t%s\t%s", __method__, pure_boolean, escape_newlines( predicate )
181
+ transmit "%s\t%s\t%s", :semantic_predicate, pure_boolean, escape_newlines( predicate )
182
182
  end
183
183
 
184
184
  def consume_node( tree )
185
- transmit "%s\t%s", __method__, serialize_node( tree )
185
+ transmit "%s\t%s", :consume_node, serialize_node( tree )
186
186
  end
187
187
 
188
188
  def adaptor
@@ -190,40 +190,40 @@ class EventSocketProxy
190
190
  end
191
191
 
192
192
  def look_tree( i, tree )
193
- transmit "%s\t%s\t%s", __method__, i, serialize_node( tree )
193
+ transmit "%s\t%s\t%s", :look_tree, i, serialize_node( tree )
194
194
  end
195
195
 
196
196
  def flat_node( tree )
197
- transmit "%s\t%i", __method__, adaptor.unique_id( tree )
197
+ transmit "%s\t%i", :flat_node, adaptor.unique_id( tree )
198
198
  end
199
199
 
200
200
  def error_node( tree )
201
- transmit "%s\t%i\t%i\t%p", __method__, adaptor.unique_id( tree ),
201
+ transmit "%s\t%i\t%i\t%p", :error_node, adaptor.unique_id( tree ),
202
202
  Token::INVALID_TOKEN_TYPE, escape_newlines( tree.to_s )
203
203
  end
204
204
 
205
205
  def create_node( node, token = nil )
206
206
  if token
207
- transmit "%s\t%i\t%i", __method__, adaptor.unique_id( node ),
207
+ transmit "%s\t%i\t%i", :create_node, adaptor.unique_id( node ),
208
208
  token.token_index
209
209
  else
210
- transmit "%s\t%i\t%i\t%p", __method__, adaptor.unique_id( node ),
210
+ transmit "%s\t%i\t%i\t%p", :create_node, adaptor.unique_id( node ),
211
211
  adaptor.type_of( node ), adaptor.text_of( node )
212
212
  end
213
213
  end
214
214
 
215
215
  def become_root( new_root, old_root )
216
- transmit "%s\t%i\t%i", __method__, adaptor.unique_id( new_root ),
216
+ transmit "%s\t%i\t%i", :become_root, adaptor.unique_id( new_root ),
217
217
  adaptor.unique_id( old_root )
218
218
  end
219
219
 
220
220
  def add_child( root, child )
221
- transmit "%s\t%i\t%i", __method__, adaptor.unique_id( root ),
221
+ transmit "%s\t%i\t%i", :add_child, adaptor.unique_id( root ),
222
222
  adaptor.unique_id( child )
223
223
  end
224
224
 
225
225
  def set_token_boundaries( t, token_start_index, token_stop_index )
226
- transmit "%s\t%i\t%i\t%i", __method__, adaptor.unique_id( t ),
226
+ transmit "%s\t%i\t%i\t%i", :set_token_boundaries, adaptor.unique_id( t ),
227
227
  token_start_index, token_stop_index
228
228
  end
229
229
 
@@ -357,4 +357,4 @@ private
357
357
 
358
358
  end # class RemoteEventSocketListener
359
359
  end # module Debug
360
- end # module ANTLR3
360
+ end # module ANTLR3
@@ -361,8 +361,9 @@ class BaseRecognizer
361
361
  # to display the error info on $stderr.
362
362
  def report_error(e = $!)
363
363
  @state.error_recovery and return
364
+ @state.syntax_errors += 1
364
365
  @state.error_recovery = true
365
- display_recognition_error(e)
366
+ display_recognition_error( e )
366
367
  end
367
368
 
368
369
  # error reporting hook for presenting the information
@@ -789,18 +790,19 @@ class BaseRecognizer
789
790
  alias backtracking backtracking_level
790
791
  alias backtracking= backtracking_level=
791
792
 
792
- def rule_memoization(rule, start_index)
793
- @state.rule_memory[rule] ||= Hash.new(MEMO_RULE_UNKNOWN)
794
- @state.rule_memory[rule][start_index]
793
+ def rule_memoization( rule, start_index )
794
+ @state.rule_memory[ rule ] ||= Hash.new( MEMO_RULE_UNKNOWN )
795
+ @state.rule_memory[ rule ][ start_index ]
795
796
  end
796
797
 
797
- def already_parsed_rule?(rule)
798
- stop_index = rule_memoization(rule, @input.index)
798
+ def already_parsed_rule?( rule )
799
+ stop_index = rule_memoization( rule, @input.index )
799
800
  case stop_index
800
801
  when MEMO_RULE_UNKNOWN then return false
801
- when MEMO_RULE_FAILED then return true
802
+ when MEMO_RULE_FAILED
803
+ raise BacktrackingFailed
802
804
  else
803
- @input.seek(stop_index + 1)
805
+ @input.seek( stop_index + 1 )
804
806
  end
805
807
  return true
806
808
  end
@@ -923,18 +925,6 @@ class Lexer < BaseRecognizer
923
925
  block_given? ? yield(main) : main.execute(argv)
924
926
  end
925
927
 
926
- def self.associated_parser
927
- @grammar_home and @grammar_home::Parser
928
- rescue NameError
929
- grammar_name = @grammar_home.name.split("::").last
930
- begin
931
- require "#{grammar_name}Parser"
932
- rescue LoadError => e
933
- return nil
934
- end
935
- return @grammar_home::Parser rescue nil
936
- end
937
-
938
928
  def self.associated_parser
939
929
  @associated_parser ||= begin
940
930
  @grammar_home and @grammar_home::Parser
@@ -949,7 +939,7 @@ class Lexer < BaseRecognizer
949
939
  end
950
940
 
951
941
  def initialize(input, options = {})
952
- super(options)
942
+ super( options )
953
943
  @input =
954
944
  case input
955
945
  when ::String then StringStream.new(input, options)
@@ -1009,7 +999,7 @@ class Lexer < BaseRecognizer
1009
999
  @input.source_name
1010
1000
  end
1011
1001
 
1012
- def emit(token = nil)
1002
+ def emit( token = @state.token )
1013
1003
  token ||= create_token
1014
1004
  @state.token = token
1015
1005
  return token
@@ -1254,7 +1244,7 @@ private
1254
1244
  def cast_input( input, options )
1255
1245
  case input
1256
1246
  when TokenSource then CommonTokenStream.new( input, options )
1257
- when IO, String
1247
+ when IO, String, CharacterStream
1258
1248
  if lexer_class = self.class.associated_lexer
1259
1249
  CommonTokenStream.new( lexer_class.new( input, options ), options )
1260
1250
  else