rltk 2.0.0 → 2.1.0

data/README.md

@@ -170,6 +170,44 @@ The default starting symbol of the grammar is the left-hand side of the first pr
 
 **Make sure you call `finalize` at the end of your parser definition, and only call it once.**
 
+### Shortcuts
+
+RLTK provides several shortcuts for common grammar constructs.  Right now these shortcuts include the {RLTK::Parser.empty_list} and {RLTK::Parser.nonempty_list} methods.  An empty list is a list that may contain 0, 1, or more elements, with a given token seperating each element.  A non-empty list contains **at least** 1 element.
+
+This example shows how these shortcuts may be used to define a list of integers separated by a `:COMMA` token:
+
+	class ListParser < RLTK::Parser
+		nonempty_list(:int_list, :INT, :COMMA)
+		
+		finalize
+	end
+
+If you wanted to define a list of floats or integers you could define your parser like this:
+
+	class ListParser < RLTK::Parser
+		nonempty_list(:mixed_list, [:INT, :FLOAT], :COMMA)
+		
+		finalize
+	end
+
+A list may also contain multiple tokens between the separator:
+
+	class ListParser < RLTK::Parser
+		nonempty_list(:foo_bar_list, 'FOO BAR', :COMMA)
+		
+		finalize
+	end
+
+Lastly, you can mix all of these features together:
+
+	class ListParser < RLTK::Parser
+		nonempty_list(:foo_list, ['FOO BAR', 'FOO BAZ+'], :COMMA)
+		
+		finalize
+	end
+
+The productions generated by these shortcuts will always evaluate to an array.  In the first two examples above the productions will produce a 1-D array containing the values of the `INT` or `FLOAT` tokens.  In the last two examples the productions `foo_bar_list` and `foo_list` will produce 2-D arrays where the top level array is composed of tuples coresponding to the values of `FOO`, and `BAR` or one or more `BAZ`s.
+
 ### Precedence and Associativity
 
 To help you remove ambiguity from your grammars RLTK lets you assign precedence and associativity information to terminal symbols.  Productions then get assigned precedence and associativity based on either the last terminal symbol on the right-hand side of the production, or an optional parameter to the {RLTK::Parser.production} or {RLTK::Parser.clause} methods.  When an {RLTK::Parser} encounters a shift/reduce error it will attempt to resolve it using the following rules:
@@ -255,6 +293,12 @@ The {RLTK::Parser.parse} and {RLTK::Parser#parse} methods also have several opti
 
 * **verbose** - Value should be `true`, `false`, an `IO` object, or a file name.  Default value is `false`.  If a non `false` (or `nil`) value is specified a detailed description of the actions of the parser are printed to $stdout, the provided `IO` object, or the specified file as it parses the input.
 
+### Parse Trees
+
+The above section briefly mentions the *parse_tree* option.  So that this neat feature doesn't get lost in the rest of the documentation here is the tree generated by the Kazoo parser from Chapter 7 of the tutorial when it parses the line `def fib(a) if a < 2 then 1 else fib(a-1) + fib(a-2);`:
+
+![Kazoo parse tree.](https://github.com/chriswailes/RLTK/raw/master/resources/simple_tree.png)
+
 ### Parsing Exceptions
 
 Calls to {RLTK::Parser.parse} may raise one of four exceptions:
@@ -268,7 +312,7 @@ Calls to {RLTK::Parser.parse} may raise one of four exceptions:
 
 **Warning: this is the lest tested feature of RLTK.  If you encounter any problems while using it, please let me know so I can fix any bugs as soon as possible.**
 
-When an RLTK parser encounters a token for which there are no more valid tokens (and it is on the last parse stack / possible parse-tree path) it will enter error handling mode.  In this mode the parser pops states and input off of the parse stack (the parser is a pushdown automaton after all) until it finds a state that has a shift action for the `ERROR` terminal.  A dummy `ERROR` terminal is then placed onto the parse stack and the shift action is taken.  This error token will have the position information of the token that caused the parser to enter error handling mode.
+When an RLTK parser encounters a token for which there are no more valid actions (and it is on the last parse stack / possible parse-tree path) it will enter error handling mode.  In this mode the parser pops states and input off of the parse stack (the parser is a pushdown automaton after all) until it finds a state that has a shift action for the `ERROR` terminal.  A dummy `ERROR` terminal is then placed onto the parse stack and the shift action is taken.  This error token will have the position information of the token that caused the parser to enter error handling mode.
 
 If the input (including the `ERROR` token) can be reduced immediately the associated error handling proc is evaluated and we continue parsing.  If the parser can't immediately reduce it will begin shifting tokens onto the input stack.  This may cause the parser to enter a state in which it again has no valid actions for an input.  When this happens it enters error handling mode again and pops states and input off of the stack until it reaches an error state again.  In this way it searches for the first substring after the error occurred for which it can resume parsing.
 
@@ -349,7 +393,7 @@ Why did I fork ruby-llvm, and why might you want to use the RLTK bindings over r
 
 * **Cleaner Codebase** - The RLTK bindings present a cleaner interface to the LLVM library by conforming to more standard Ruby programming practices, providing better abstractions and cleaner inheritance hierarchies, overloading constructors and other methods properly, and performing type checking on objects to better aid in debugging.
 * **Documentation** - RLTK's bindings provide slightly better documentation.
-* **Completeness** - The RLTK bindings provide several features that are missing from the ruby-llvm project.  These include the ability to initialize LLVM for architectures besides x86 (RLTK supports all architectures supported by LLVM) and the presence of all of LLVM's optimization passes.
+* **Completeness** - The RLTK bindings provide several features that are missing from the ruby-llvm project.  These include the ability to initialize LLVM for architectures besides x86 (RLTK supports all architectures supported by LLVM), the presence of all of LLVM's optimization passes, and the ability to print the LLVM IR representation of modules and values to files.
 * **Ease of Use** - Several features have been added to make generating code easier.
 * **Speed** - The RLTK bindings are ever so slightly faster due to avoiding unnecessary FFI calls.
 

data/Rakefile

@@ -137,7 +137,16 @@ task :gen_bindings do
 	headers = [
 		'llvm-ecb.h',
 		
-		'llvm-ecb/support.h'
+		'llvm-ecb/asm.h',
+		'llvm-ecb/module.h',
+		'llvm-ecb/support.h',
+		
+		'llvm-ecb/value.h',
+		'llvm-ecb/target.h'
+
+		# This causes value.h to not be included.
+		#'llvm-ecb/target.h',
+		#'llvm-ecb/value.h'
 	]
 	
 	begin

data/lib/rltk/ast.rb

@@ -272,11 +272,52 @@ module RLTK # :nodoc:
 			@notes.delete(key)
 		end
 		
-		# An iterator over the node's children.
+		# This method is a simple wrapper around Marshal.dump, and is used
+		# to serialize an AST.  You can use Marshal.load to reconstruct a
+		# serialized AST.
+		#
+		# @param [nil, IO, String]	dest		Where the serialized version of the AST will end up.  If nil, this method will return the AST as a string.
+		# @param [Fixnum]			limit	Recursion depth.  If -1 is specified there is no limit on the recursion depth.
+		#
+		# @return [void, String] String if *dest* is nil, void otherwise.
+		def dump(dest = nil, limit = -1)
+			case dest
+			when nil		then Marshal.dump(self, limit)
+			when String	then File.open(dest, 'w') { |f| Marshal.dump(self, f, limit) }
+			when IO		then Marshal.dump(self, dest, limit)
+			else	raise TypeError, "AST#dump expects nil, a String, or an IO object for the dest parameter."
+			end
+		end
+		
+		# An iterator over the node's children.  The AST may be traversed in
+		# the following orders:
+		#
+		# * Pre-order (:pre)
+		# * Post-order (:post)
+		# * Level-order (:level)
 		#
 		# @return [void]
-		def each
-			self.children.each { |c| yield c }
+		def each(order = :pre, &block)
+			case order
+			when :pre
+				yield self
+				
+				self.children.compact.each { |c| c.each(:pre, &block) }
+				
+			when :post
+				self.children.compact.each { |c| c.each(:post, &block) }
+				
+				yield self
+				
+			when :level
+				level_queue = [self]
+				
+				while node = level_queue.shift
+					yield node
+					
+					level_queue += node.children.compact
+				end
+			end
 		end
 		
 		# Tests to see if a note named *key* is present at this node.
@@ -300,6 +341,10 @@ module RLTK # :nodoc:
 				@notes	= Hash.new()
 				@parent	= nil
 				
+				# Pad out the objects array with nil values.
+				max_args = self.class.value_names.length + self.class.child_names.length
+				objects.fill(nil, objects.length...max_args)
+				
 				pivot = self.class.value_names.length
 				
 				self.values	= objects[0...pivot]

data/lib/rltk/cfg.rb

@@ -96,11 +96,13 @@ module RLTK # :nodoc:
 		# @return [void]
 		def add_production(production)
 			@productions_sym[production.lhs] << (@productions_id[production.id] = production)
+			
+			production
 		end
 		
 		# Sets the EBNF callback to *callback*.
 		#
-		# @param [Proc] callback A Proc object to be called when EBNF operators are expanded.
+		# @param [Proc] callback A Proc object to be called when EBNF operators are expanded and list productions are added.
 		#
 		# @return [void]
 		def callback(&callback)
@@ -116,9 +118,7 @@ module RLTK # :nodoc:
 		#
 		# @return [Production]
 		def clause(expression)
-			if not @curr_lhs
-				raise GrammarError, 'CFG.clause called outside of CFG.production block.'
-			end
+			raise GrammarError, 'CFG#clause called outside of CFG#production block.' if not @curr_lhs
 			
 			lhs		= @curr_lhs.to_sym
 			rhs		= Array.new
@@ -144,17 +144,10 @@ module RLTK # :nodoc:
 						
 						rhs <<
 						case ttype1
-							when :'?'
-								self.get_question(tvalue0)
-							
-							when :*
-								self.get_star(tvalue0)
-							
-							when :+
-								self.get_plus(tvalue0)
-							
-							else
-								tvalue0
+						when :'?'	then self.get_question(tvalue0)						
+						when :*	then self.get_star(tvalue0)
+						when :+	then self.get_plus(tvalue0)
+						else			tvalue0
 						end
 					else
 						rhs << tvalue0
@@ -174,6 +167,34 @@ module RLTK # :nodoc:
 			return production
 		end
 		
+		# This method adds the necessary productions for empty lists to the
+		# grammar.  These productions are named `symbol`, `symbol + '_prime'`
+		# and `symbol + '_elements'`
+		#
+		# @param [Symbol]		symbol		The name of the production to add.
+		# @param [Array<String>]	list_elements	An array of expressions that may appear in the list.
+		# @param [Symbol]		separator		The list separator symbol.
+		#
+		# @return [void]
+		def empty_list_production(symbol, list_elements, separator)
+			# Add the items for the following productions:
+			#
+			# symbol: | symbol_prime
+			
+			prime = symbol.to_s + '_prime'
+			
+			# 1st Production
+			production = self.production(symbol, '').first
+			@callback.call(production, :elp, :first)
+			
+			# 2nd Production
+			production = self.production(symbol, prime.to_s).first
+			@callback.call(production, :elp, :second)
+			
+			self.nonempty_list(prime, list_elements, separator)
+		end
+		alias :empty_list :empty_list_production
+		
 		# @param [Symbol, Array<Symbol>] sentence Sentence to find the *first set* for.
 		#
 		# @return [Array<Symbol>] The *first set* for the given sentence.
@@ -310,11 +331,11 @@ module RLTK # :nodoc:
 				# token_plus: token | token token_plus
 				
 				# 1st production
-				self.add_production(production = Production.new(self.next_id, new_symbol, [symbol]))
+				production = self.add_production(Production.new(self.next_id, new_symbol, [symbol]))
 				@callback.call(production, :+, :first)
 				
 				# 2nd production
-				self.add_production(production = Production.new(self.next_id, new_symbol, [symbol, new_symbol]))
+				production = self.add_production(Production.new(self.next_id, new_symbol, [new_symbol, symbol]))
 				@callback.call(production, :+, :second)
 				
 				# Add the new symbol to the list of nonterminals.
@@ -338,11 +359,11 @@ module RLTK # :nodoc:
 				# nonterm_question: | nonterm
 				
 				# 1st (empty) production.
-				self.add_production(production = Production.new(self.next_id, new_symbol, []))
+				production = self.add_production(Production.new(self.next_id, new_symbol, []))
 				@callback.call(production, :'?', :first)
 				
 				# 2nd production
-				self.add_production(production = Production.new(self.next_id, new_symbol, [symbol]))
+				production = self.add_production(Production.new(self.next_id, new_symbol, [symbol]))
 				@callback.call(production, :'?', :second)
 				
 				# Add the new symbol to the list of nonterminals.
@@ -366,11 +387,11 @@ module RLTK # :nodoc:
 				# token_star: | token token_star
 				
 				# 1st (empty) production
-				self.add_production(production = Production.new(self.next_id, new_symbol, []))
+				production = self.add_production(Production.new(self.next_id, new_symbol, []))
 				@callback.call(production, :*, :first)
 				
 				# 2nd production
-				self.add_production(production = Production.new(self.next_id, new_symbol, [symbol, new_symbol]))
+				production = self.add_production(Production.new(self.next_id, new_symbol, [new_symbol, symbol]))
 				@callback.call(production, :*, :second)
 				
 				# Add the new symbol to the list of nonterminals.
@@ -385,6 +406,54 @@ module RLTK # :nodoc:
 			@production_counter += 1
 		end
 		
+		# This method adds the necessary productions for non-empty lists to
+		# the grammar.  These productions are named `symbol` and
+		# `symbol + '_elements'`
+		#
+		# @param [Symbol]						symbol		The name of the production to add.
+		# @param [String, Symbol, Array<String>]	list_elements	Expression(s) that may appear in the list.
+		# @param [Symbol]						separator		The list separator symbol.
+		#
+		# @return [void]
+		def nonempty_list_production(symbol, list_elements, separator)
+			# Add the items for the following productions:
+			#
+			# symbol_elements: list_elements.join('|')
+			#
+			# symbol: symbol_elements | symbol separator symbol_elements
+			
+			if list_elements.is_a?(String) or list_elements.is_a?(Symbol)
+				list_elements = [list_elements.to_s]
+				
+			elsif list_elements.is_a?(Array)
+				if list_elements.empty?
+					raise ArgumentError, 'Parameter list_elements must not be empty.'
+				else
+					list_elements.map! { |el| el.to_s }
+				end
+				
+			else
+				raise ArgumentError, 'Parameter list_elements must be a String, Symbol, or array of Strings and Symbols.'
+			end
+			
+			el_symbol = (symbol.to_s + '_elements').to_sym
+			
+			# 1st Production
+			production = self.production(symbol, el_symbol.to_s).first
+			@callback.call(production, :nelp, :first)
+			
+			# 2nd Production
+			production = self.production(symbol, "#{symbol} #{separator} #{el_symbol}").first
+			@callback.call(production, :nelp, :second)
+			
+			# 3rd Productions
+			list_elements.each do |el|
+				production = self.production(el_symbol, el).first
+				@callback.call(production, :nelp, :third)
+			end
+		end
+		alias :nonempty_list :nonempty_list_production
+		
 		# @return [Array<Symbol>] All terminal symbols used in the grammar's definition.
 		def nonterms
 			@nonterms.keys

data/lib/rltk/cg/bindings.rb

@@ -74,23 +74,47 @@ module RLTK::CG # :nodoc:
 		
 		# List of architectures supported by LLVM.
 		ARCHS = [
-			:ARM,
 			:Alpha,
+			:ARM,
 			:Blackfin,
 			:CBackend,
 			:CellSPU,
 			:CppBackend,
 			:MBlaze,
-			:MSP430,
 			:Mips,
-			:PTX,
+			:MSP430,
 			:PowerPC,
+			:PTX,
 			:Sparc,
 			:SystemZ,
-			:XCore,
+			:X86,
+			:XCore
+		]
+		
+		# List of assembly parsers.
+		ASM_PARSERS = [
+			:ARM,
+			:MBLaze,
 			:X86
 		]
-	
+		
+		# List of assembly printers.
+		ASM_PRINTERS = [
+			:Alpha,
+			:ARM,
+			:Blackfin,
+			:CellSPU,
+			:MBLaze,
+			:Mips,
+			:MSP430,
+			:PowerPC,
+			:PTX,
+			:Sparc,
+			:SystemZ,
+			:X86,
+			:XCore
+		]
+		
 		###########
 		# Methods #
 		###########
@@ -131,6 +155,14 @@ module RLTK::CG # :nodoc:
 			add_binding("LLVMInitialize#{arch}TargetMC", [], :void)
 		end
 		
+		ASM_PARSERS.each do |asm|
+			add_binding("LLVMInitialize#{asm}AsmParser", [], :void)
+		end
+		
+		ASM_PRINTERS.each do |asm|
+			add_binding("LLVMInitialize#{asm}AsmPrinter", [], :void)
+		end
+		
 		add_binding(:LLVMDisposeMessage, [:pointer], :void)
 	end
 end

data/lib/rltk/cg/execution_engine.rb

@@ -12,6 +12,7 @@
 require 'rltk/util/abstract_class'
 require 'rltk/cg/bindings'
 require 'rltk/cg/pass_manager'
+require 'rltk/cg/target'
 
 #######################
 # Classes and Modules #
@@ -38,6 +39,8 @@ module RLTK::CG # :nodoc:
 		#
 		# @raise [RuntimeError] An error is raised if something went horribly wrong inside LLVM during the creation of this engine.
 		def initialize(mod, &block)
+			check_type(mod, Module, 'mod')
+			
 			block = Proc.new { |ptr, error| Bindings.create_execution_engine_for_module(ptr, mod, error) } if block == nil
 			
 			ptr		= FFI::MemoryPointer.new(:pointer)
@@ -47,6 +50,9 @@ module RLTK::CG # :nodoc:
 			if status.zero?
 				@ptr		= ptr.read_pointer
 				@module	= mod
+				
+				# Associate this engine with the provided module.
+				@module.engine = self
 		
 			else
 				errorp  = error.read_pointer
@@ -71,18 +77,6 @@ module RLTK::CG # :nodoc:
 			end
 		end
 		
-		# @return [FunctionPassManager] Function pass manager for this engine.
-		def function_pass_manager
-			@function_pass_manager ||= FunctionPassManager.new(self, @module)
-		end
-		alias :fpm :function_pass_manager
-		
-		# @return [PassManager] Pass manager for this engine.
-		def pass_manager
-			@pass_manager ||= PassManager.new(self, @module)
-		end
-		alias :pm :pass_manager
-		
 		# Builds a pointer to a global value.
 		#
 		# @param [GlobalValue] global Value you want a pointer to.
@@ -141,6 +135,11 @@ module RLTK::CG # :nodoc:
 			GenericValue.new(Bindings.run_function_as_main(@ptr, fun, args.length, argv, env))
 		end
 		alias :run_main :run_function_as_main
+		
+		# @return [TargetData] Information about the target architecture for this execution engine.
+		def target_data
+			TargetData.new(Bindings.get_execution_engine_target_data(@ptr))
+		end
 	end
 	
 	# An execution engine that interprets the given code.

data/lib/rltk/cg/generated_bindings.rb

@@ -5102,16 +5102,10 @@ module RLTK::CG::Bindings
   # function returns the number of bytes in the instruction or zero if there was
   # no valid instruction.
   # 
-  # @method disasm_instruction(dc, bytes, bytes_size, pc, out_string, out_string_size)
-  # @param [FFI::Pointer(DisasmContextRef)] dc 
-  # @param [FFI::Pointer(*Uint8T)] bytes 
-  # @param [Integer] bytes_size 
-  # @param [Integer] pc 
-  # @param [String] out_string 
-  # @param [Integer] out_string_size 
+  # @method disasm_instruction()
   # @return [Integer] 
   # @scope class
-  attach_function :disasm_instruction, :LLVMDisasmInstruction, [:pointer, :pointer, :ulong, :ulong, :string, :ulong], :ulong
+  attach_function :disasm_instruction, :LLVMDisasmInstruction, [], :int
   
   # (Not documented)
   #