antlr3 1.4.0 → 1.6.0

data/History.txt

@@ -1,66 +1,59 @@
-h1(update). V1.1.1 2009-10-24 Kyle Yetter  <kcy5b@yahoo.com>
-
-h2(minor). minor alteration to TokenScheme
-
-* TokenScheme#register_name would raise an error if the token type was anonymous (i.e. T__19 or such) and the new name didn't have the format "'a string'". That works well for ANTLR tokens, but it's generally pointless, so I got rid of those restrictions. 
-
-|\2. MODIFIED |
-| lib/antlr3/token.rb | made minor changes to ANTLR3::TokenScheme#register_name |
-
-h1(update). V1.1.0 2009-10-19 Kyle Yetter  <kcy5b@yahoo.com>
-
-h2(major). moved FilterMode
-
-* since <tt>ANTLR3::Lexer::FilterMode</tt> is only necessary in lexers that specify filter=true, I moved it out of antlr3/recognizers.rb to its own file, antlr3/modes/filter.rb.
-
-* the FilterMode module is no longer in the Lexer namespace, but in the broader ANTLR3 namespace ... i.e. module ANTLR3::Lexer::FilterMode ==> module ANTLR3::FilterMode
-
-* I added an autoload entry to the ANTLR3 module in antlr3.rb for FilterMode, thus the module is autoloaded as needed whenever the constant ANTLR3::FilterMode is referenced
-
-|\2. MODIFIED                                                       |
-| lib/antlr3/recognizers.rb  | removed Lexer::FilterMode definition |
-| lib/antlr3.rb              | added autoload :FilterMode entry     |
-|\2. CREATED                                                        |
-| lib/antlr3/modes/filter.rb | contains FilterMode module extracted from recognizers.rb |
-
-h2(major). new module ASTBuilder
-  
-* created a new module, ANTLR3::ASTBuilder
-* all tree-building recognizers (parsers and tree parsers with the output=AST option) include this module
-* provides a way to test a recognizer to see if it is an AST-building recognizer (rec.is_a?(ANTLR3::ASTBuilder)) and it also added a number of methods to help clean up generated AST-parser code
-
-|\2. MODIFIED                                                       |
-| lib/antlr3.rb              | added the usual autoload entry for ASTBuilder |
-| templates/AST.stg          | added ``include ANTLR3::ASTBuilder'' to automatically include the module on all AST-output recognizers |
-|\2. CREATED                                                        |
-| lib/antlr3/modes/ast-builder.rb | defined new module ANTLR3::ASTBuilder |
-  
-h2(major). refactored debug and profile modes
-  
-* began process of organizing/cleaning-up the ANTLR3::Debug module (in file lib/antlr3/debug.rb), which contains all runtime support code and features for --debug and --profile option code output.
-* extracted profiling-oriented classes and modules from the Debug module and moved them to a new file, lib/antlr3/profile.rb.
-* since usually you're only using one particular type of Debug::EventListener at a time, I moved the six or so different EventListener classes out of Debug and into their own respective files in the antlr3/debug directory
-* since tree-oriented debugging hooks (Debug::TreeAdaptor and such) are only necessary for tree-oriented code, and thus are not part of the essential core, I moved the tree-related event modules out of Debug and into a separate file, lib/antlr3/tree/debug.rb
-* debug.rb now only defines the core EventListener interface module and the event-hook modules ParserEvents, TokenStream, and the expanded RecognizerSharedState class
-* as with most classes that aren't loaded by default in the runtime library, I strategically placed autoload statements in appropriate modules to autoload any of the Debug modules and classes I yanked out of the debug.rb, so there shouldn't be any need for extra 'require' statements to use the classes
-
-|\2. MODIFIED                                                       |
-| lib/antlr3/debug.rb | removed definitions: EventSocketProxy, RemoteEventSocketListener, TraceEventListener, RuleTracer, EventHub, TreeNodeStream, TreeAdaptor, ParserProfilingEvents, Profiler, Profiler::Profile, Profiler::DataSet |
-|\2. CREATED                                                        |
-| lib/antlr3/debug/event-hub.rb | new home for Debug::EventHub |
-| lib/antlr3/debug/record-event-listener.rb | new home for Debug::RecordEventListener |
-| lib/antlr3/debug/rule-tracer.rb | new home for Debug::RuleTracer |
-| lib/antlr3/debug/socket.rb | new home for Debug::EventSocketProxy and Debug::RemoteEventSocketListener |
-| lib/antlr3/debug/trace-event-listener.rb | new home for Debug::TraceEventListener |
-| lib/antlr3/profile.rb | new profile-oriented module, Profile, contains former debug classes Profile, ProfileEvents, and Profiler |
-| lib/antlr3/tree/debug.rb | new home for Debug::TreeAdaptor and Debug::TreeNodeStream |
-
-h2(minor). moved most of the core-extension code to new Util module
-
-* as handy as it is to add new functions to core classes, I moved most of the code in the core-extension directory to the new slightly-nebulous, catch-all dumping ground modules, ANTLR3::Util and ANTLR3::ClassMacros. String and Module methods defined in the extensions were only used in a small handful of places and thus it seemed intrusive to polute the core classes with these utility methods for little payoff.
-* ANTLR classes now extend ANTLR3::ClassMacros when necessary. It defines 'abstract' to easily create abstract methods. It also provides a new method, #shared_attribute, which defines an attribute accessor with a writer
-
-h1(update). V1.0.0 2009-10-14 Kyle Yetter  <kcy5b@yahoo.com>
-
-h2(major). Initial Release: Birthday!
-
+=== 1.6.0 / 01-27-10
+
+* 4 Minor Enhancements to runtime library
+  - CommonTokenStream and subclasses can now also accept another CommonTokenStream
+    object, serving sort of as a copy constructor. This provides a reasonable way
+    to cast a CommonTokenStream into a TokenRewriteStream and vice-versa
+  - CommonTreeNodeStream can take an options hash at the end of any of its various
+    signatures to provide values for attributes like token_stream
+  - If a parser is created with a CharacterStream object (i.e., StringStream), it
+    will also try to cast it by checking for an associated lexer (like it currently
+    does when created with String or IO objects).
+  - Template-output mode parsers have a new class method "load_templates( path )"
+    which will attempt to load the template group file at +path+ and, if successful,
+    set it to the parser's default template library. (See the CMinus example set for
+    an example of how this can be used)
+
+* 2 bug fixes / 1 change to templates
+  - backtracking was broken in situations where recognition failed when
+    a semantic predicate was encountered as the template code did not
+    throw BacktrackingFailed when it was in backtracking mode instead of
+    PredicateFailed
+  - rule memoization also had a major bug in that rules that were memoized
+    after failure did not throw BacktrackingFailed when it was needed
+  - I changed `attribute scope' structures (not rule return scopes, but
+    the groups of attributes you can assign to rules with the `scope' keyword),
+    from being implemented as named constants in the class body to class
+    variables belonging to the recognizer. As class variables do not need
+    to follow naming conventions, this provides more freedom in naming without
+    all of the ugly name mangling. These are generally used privately by parsers,
+    so it shouldn't be too drastic of a change as to break end user's code.
+
+=== 1.5.0 / 01-25-10
+
+* 2 minor enhancements
+  - Added BaseTree#walk, which iterates in a top-down fashion across all notes in a tree.
+  - Added BaseTree#prune, which stops decent into the current node during BaseTree#walk.
+
+* 1 optimization
+  - changed `closure loops' ( uses of the `*' or `+' modifiers ) from using
+    Kernel#loop to `while true ... end' style loops, which should be slightly
+    faster
+
+* new test code
+  - test/functiona/template-output/template-output.rb serves as a basic
+    functional test implementation for the new template output mode
+
+=== 1.4.0 / 01-17-10
+
+* 1 major new feature
+  - finished basic implementation of a template output mode.
+  - lib/antlr3/template.rb defines a number of classes for
+    ERB-based template output translation
+  - templates/ST.stg defines the Ruby target templates to
+    generate template output mode recognizers
+    
+* new test code
+  - added template unit tests at test/unit/test-template.rb
+  - test/functional/template-output/template-output.rb serves as a basic
+    functional test implementation for the new template output mode

data/java/RubyTarget.java

@@ -262,6 +262,7 @@ extends Target
 			generator.getRecognizerFileName(grammar.name, grammar.type);
 		generator.write(outputFileST, fileName);
 	}
+	
 	public String getTargetCharLiteralFromANTLRCharLiteral(
 			CodeGenerator generator,
 			String literal)
@@ -285,11 +286,13 @@ extends Target
 
 		return result;
 	}
+	
 	public int getMaxCharValue(CodeGenerator generator)
 	{
-		// we don't support unicode, yet.
+		// Versions before 1.9 do not support unicode
 		return 0xFF;
 	}
+	
 	public String getTokenTypeAsTargetLabel(CodeGenerator generator, int ttype)
 	{
 		String name = generator.grammar.getTokenDisplayName(ttype);
@@ -299,6 +302,7 @@ extends Target
 		}
 		return name;
 	}
+	
 	/** Is scope in @scope::name {action} valid for this kind of grammar?
 	 *  Targets like C++ may want to allow new scopes like headerfile or
 	 *  some such.  The action names themselves are not policed at the
@@ -369,6 +373,7 @@ extends Target
 		}
 		return false;
 	}
+	
 	/*
     public String getTargetStringLiteralFromString(String s)
     {
@@ -391,17 +396,17 @@ extends Target
         System.out.print(result + "\n");
         return result;
     }
-	 */
+	*/
+	
 	public String encodeIntAsCharEscape(final int v) {
 		final int intValue;
 		
-		if (v == 65535) {
+		if ( v == 65535 ) {
 		  intValue = -1;
 		} else {
 		  intValue = v;
 		}
 		
-		return String.valueOf(intValue);
-		// return String.valueOf(v);
+		return String.valueOf( intValue );
 	}
 }

data/lib/antlr3/debug.rb

@@ -228,21 +228,16 @@ module ParserEvents
     @state.backtracking -= 1
   end
   
-  def report_error(exc)
-    case exc
-    when ANTLR3::Error::RecognitionError
-      @debug_listener.recognition_exception(exc)
-    else
-      $stderr.puts(exc)
-      $stderr.puts(exc.backtrace)
-    end
+  def report_error( exc )
+    ANTLR3::RecognitionError === exc and
+      @debug_listener.recognition_exception( exc )
     super
   end
   
-  def missing_symbol(error, expected_type, follow)
+  def missing_symbol( error, expected_type, follow )
     symbol = super
-    @debug_listener.consume_node(symbol)
-    return(symbol)
+    @debug_listener.consume_node( symbol )
+    return( symbol )
   end
   
   def in_rule(grammar_file, rule_name)
@@ -363,11 +358,10 @@ module TokenStream
     return @last_marker
   end
   
-  def rewind(marker = nil)
-    @debug_listener.rewind(marker)
+  def rewind( marker = nil, release = true )
+    @debug_listener.rewind( marker )
     super
   end
-  
 end
 
 =begin rdoc ANTLR3::Debug::EventListener

data/lib/antlr3/debug/socket.rb

@@ -99,39 +99,39 @@ class EventSocketProxy
   end
   
   def enter_rule( grammar_file_name, rule_name )
-    transmit "%s\t%s\t%s", __method__, grammar_file_name, rule_name
+    transmit "%s\t%s\t%s", :enter_rule, grammar_file_name, rule_name
   end
   
   def enter_alternative( alt )
-    transmit "%s\t%s", __method__, alt
+    transmit "%s\t%s", :enter_alternative, alt
   end
   
   def exit_rule( grammar_file_name, rule_name )
-    transmit "%s\t%s\t%s", __method__, grammar_file_name, rule_name
+    transmit "%s\t%s\t%s", :exit_rule, grammar_file_name, rule_name
   end
   
   def enter_subrule( decision_number )
-    transmit "%s\t%i", __method__, decision_number
+    transmit "%s\t%i", :enter_subrule, decision_number
   end
   
   def exit_subrule( decision_number )
-    transmit "%s\t%i", __method__, decision_number
+    transmit "%s\t%i", :exit_subrule, decision_number
   end
   
   def enter_decision( decision_number )
-    transmit "%s\t%i", __method__, decision_number
+    transmit "%s\t%i", :enter_decision, decision_number
   end
   
   def exit_decision( decision_number )
-    transmit "%s\t%i", __method__, decision_number
+    transmit "%s\t%i", :exit_decision, decision_number
   end
   
   def consume_token( token )
-    transmit "%s\t%s", __method__, serialize_token( token )
+    transmit "%s\t%s", :consume_token, serialize_token( token )
   end
   
   def consume_hidden_token( token )
-    transmit "%s\t%s", __method__, serialize_token( token )
+    transmit "%s\t%s", :consume_hidden_token, serialize_token( token )
   end
   
   def look(i, item)
@@ -140,49 +140,49 @@ class EventSocketProxy
       look_tree( i, item )
     when nil
     else
-      transmit "%s\t%i\t%s", __method__, i, serialize_token(item)
+      transmit "%s\t%i\t%s", :look, i, serialize_token(item)
     end
   end
   
   def mark(i)
-    transmit "%s\t%i", __method__, i
+    transmit "%s\t%i", :mark, i
   end
   
   def rewind(i = nil)
-    i ? transmit( "%s\t%i", __method__, i ) : transmit( '%s', __method__ )
+    i ? transmit( "%s\t%i", :rewind, i ) : transmit( '%s', :rewind )
   end
   
   def begin_backtrack( level )
-    transmit "%s\t%i", __method__, level
+    transmit "%s\t%i", :begin_backtrack, level
   end
   def end_backtrack( level, successful )
-    transmit "%s\t%i\t%p", __method__, level, (successful ? true : false)
+    transmit "%s\t%i\t%p", :end_backtrack, level, (successful ? true : false)
   end
   
   def location( line, position )
-    transmit "%s\t%i\t%i", __method__, line, position
+    transmit "%s\t%i\t%i", :location, line, position
   end
   
   def recognition_exception( exception )
-    transmit "%s\t%p\t%i\t%i\t%i", __method__, exception.class,
+    transmit "%s\t%p\t%i\t%i\t%i", :recognition_exception, exception.class,
       exception.index, exception.line, exception.column
   end
   
   def begin_resync
-    transmit '%s', __method__
+    transmit '%s', :begin_resync
   end
   
   def end_resync
-    transmit '%s', __method__
+    transmit '%s', :end_resync
   end
   
   def semantic_predicate( result, predicate )
     pure_boolean = !(!result)
-    transmit "%s\t%s\t%s", __method__, pure_boolean, escape_newlines( predicate )
+    transmit "%s\t%s\t%s", :semantic_predicate, pure_boolean, escape_newlines( predicate )
   end
   
   def consume_node( tree )
-    transmit "%s\t%s", __method__, serialize_node( tree )
+    transmit "%s\t%s", :consume_node, serialize_node( tree )
   end
   
   def adaptor
@@ -190,40 +190,40 @@ class EventSocketProxy
   end
   
   def look_tree( i, tree )
-    transmit "%s\t%s\t%s", __method__, i, serialize_node( tree )
+    transmit "%s\t%s\t%s", :look_tree, i, serialize_node( tree )
   end
   
   def flat_node( tree )
-    transmit "%s\t%i", __method__, adaptor.unique_id( tree )
+    transmit "%s\t%i", :flat_node, adaptor.unique_id( tree )
   end
   
   def error_node( tree )
-    transmit "%s\t%i\t%i\t%p", __method__, adaptor.unique_id( tree ),
+    transmit "%s\t%i\t%i\t%p", :error_node, adaptor.unique_id( tree ),
             Token::INVALID_TOKEN_TYPE, escape_newlines( tree.to_s )
   end
   
   def create_node( node, token = nil )
     if token
-      transmit "%s\t%i\t%i", __method__, adaptor.unique_id( node ),
+      transmit "%s\t%i\t%i", :create_node, adaptor.unique_id( node ),
               token.token_index
     else
-      transmit "%s\t%i\t%i\t%p", __method__, adaptor.unique_id( node ),
+      transmit "%s\t%i\t%i\t%p", :create_node, adaptor.unique_id( node ),
           adaptor.type_of( node ), adaptor.text_of( node )
     end
   end
   
   def become_root( new_root, old_root )
-    transmit "%s\t%i\t%i", __method__, adaptor.unique_id( new_root ),
+    transmit "%s\t%i\t%i", :become_root, adaptor.unique_id( new_root ),
               adaptor.unique_id( old_root )
   end
   
   def add_child( root, child )
-    transmit "%s\t%i\t%i", __method__, adaptor.unique_id( root ),
+    transmit "%s\t%i\t%i", :add_child, adaptor.unique_id( root ),
              adaptor.unique_id( child )
   end
   
   def set_token_boundaries( t, token_start_index, token_stop_index )
-    transmit "%s\t%i\t%i\t%i", __method__, adaptor.unique_id( t ),
+    transmit "%s\t%i\t%i\t%i", :set_token_boundaries, adaptor.unique_id( t ),
                                token_start_index, token_stop_index
   end
   
@@ -357,4 +357,4 @@ private
   
 end # class RemoteEventSocketListener
 end # module Debug
-end # module ANTLR3
+end # module ANTLR3

data/lib/antlr3/recognizers.rb

@@ -361,8 +361,9 @@ class BaseRecognizer
   # to display the error info on $stderr. 
   def report_error(e = $!)
     @state.error_recovery and return
+    @state.syntax_errors += 1
     @state.error_recovery = true
-    display_recognition_error(e)
+    display_recognition_error( e )
   end
   
   # error reporting hook for presenting the information
@@ -789,18 +790,19 @@ class BaseRecognizer
   alias backtracking backtracking_level
   alias backtracking= backtracking_level=
   
-  def rule_memoization(rule, start_index)
-    @state.rule_memory[rule] ||= Hash.new(MEMO_RULE_UNKNOWN)
-    @state.rule_memory[rule][start_index]
+  def rule_memoization( rule, start_index )
+    @state.rule_memory[ rule ] ||= Hash.new( MEMO_RULE_UNKNOWN )
+    @state.rule_memory[ rule ][ start_index ]
   end
   
-  def already_parsed_rule?(rule)
-    stop_index = rule_memoization(rule, @input.index)
+  def already_parsed_rule?( rule )
+    stop_index = rule_memoization( rule, @input.index )
     case stop_index
     when MEMO_RULE_UNKNOWN then return false
-    when MEMO_RULE_FAILED then return true
+    when MEMO_RULE_FAILED
+      raise BacktrackingFailed
     else
-      @input.seek(stop_index + 1)
+      @input.seek( stop_index + 1 )
     end
     return true
   end
@@ -923,18 +925,6 @@ class Lexer < BaseRecognizer
     block_given? ? yield(main) : main.execute(argv)
   end
   
-  def self.associated_parser
-    @grammar_home and @grammar_home::Parser
-  rescue NameError
-    grammar_name = @grammar_home.name.split("::").last
-    begin
-      require "#{grammar_name}Parser"
-    rescue LoadError => e
-      return nil
-    end
-    return @grammar_home::Parser rescue nil
-  end
-
   def self.associated_parser
     @associated_parser ||= begin
       @grammar_home and @grammar_home::Parser
@@ -949,7 +939,7 @@ class Lexer < BaseRecognizer
   end
 
   def initialize(input, options = {})
-    super(options)
+    super( options )
     @input =
       case input
       when ::String then StringStream.new(input, options)
@@ -1009,7 +999,7 @@ class Lexer < BaseRecognizer
     @input.source_name
   end
   
-  def emit(token = nil)
+  def emit( token = @state.token )
     token ||= create_token
     @state.token = token
     return token
@@ -1254,7 +1244,7 @@ private
   def cast_input( input, options )
     case input
     when TokenSource then CommonTokenStream.new( input, options )
-    when IO, String
+    when IO, String, CharacterStream
       if lexer_class = self.class.associated_lexer
         CommonTokenStream.new( lexer_class.new( input, options ), options )
       else