parslet 0.9.0 → 0.10.1

data/Gemfile

@@ -1,7 +1,11 @@
 # A sample Gemfile
 source "http://rubygems.org"
 
+gem 'blankslate', '>= 2.1.2.3'
+
 group :development do
   gem 'rspec'
   gem 'flexmock'
+  
+  gem 'sdoc'
 end

data/HISTORY.txt

@@ -1,4 +1,27 @@
-= 0.9.0 / ???
+= 0.10.1 / ???
+
+  + Allow match['a-z'], shortcut for match('[a-z]')
+
+  ! Fixed output inconsistencies (behaviour in connection to 'maybe')
+
+= 0.10.0 / 22Nov2010
+
+  + Parslet::Transform now takes a block on initialisation, wherein you can
+    define all the rules directly.
+    
+  + Parslet::Transform now only passes a hash to the block during transform
+    when its arity is 1. Otherwise all hash contents as bound as local     
+    variables.
+    
+  + Both inline and other documentation have been improved. 
+  
+  + You can now use 'subtree(:x)' to bind any subtree to x during tree pattern
+    matching. 
+    
+  + Transform classes can now include rules into class definition. This makes
+    Parser and Transformer behave the same. 
+  
+= 0.9.0 / 28Oct2010
   * More of everything: Examples, documentation, etc...
 
   * Breaking change: Ruby's binary or ('|') is now used for alternatives, 

data/README

@@ -1,48 +1,17 @@
 INTRODUCTION
 
-A small library that implements a PEG grammar. PEG means Parsing Expression
-Grammars [1]. These are a different kind of grammars that recognize almost the
-same languages as your conventional LR parser, except that they are easier to
-work with, since they haven't been conceived for generation, but for
-recognition of languages. You can read the founding paper of the field by
-Bryan Ford here [2].
-
-Other Ruby projects that work on the same topic are: 
-http://wiki.github.com/luikore/rsec/
-http://github.com/mjijackson/citrus
-http://github.com/nathansobo/treetop
-
-My goal here was to see how a parser/parser generator should be constructed to
-allow clean AST construction and good error handling. It seems to me that most
-often, parser generators only handle the success-case and forget about
-debugging and error generation.
-
-More specifically, this library is motivated by one of my compiler projects. I
-started out using 'treetop' (see the link above), but found it unusable. It
-was lacking in
-
-  * error reporting: Hard to see where a grammar fails. 
-  
-  * stability of generated trees: Intermediary trees were dictated by the 
-    grammar. It was hard to define invariants in that system - what was 
-    convenient when writing the grammar often wasn't in subsequent stages. 
-    
-  * clarity of parser code: The parser code is generated and is very hard
-    to read. Add that to the first point to understand my pain. 
-    
-So parslet tries to be different. It doesn't generate the parser, but instead
-defines it in a DSL which is very close to what you find in [2]. A successful
-parse then generates a parser tree consisting entirely of hashes and arrays
-and strings (read: instable). This parser tree can then be converted to a real
-AST (read: stable) using a pattern matcher that is also part of this library.
-
-Error reporting is another area where parslet excels: It is able to print not
-only the error you are used to seeing ('Parse failed because of REASON at line
-1 and char 2'), but also prints what led to that failure in the form of a
-tree (#error_tree method).
-
-[1] http://en.wikipedia.org/wiki/Parsing_expression_grammar
-[2] http://pdos.csail.mit.edu/~baford/packrat/popl04/peg-popl04.pdf
+Parslet makes developing complex parsers easy. It does so by
+
+* providing the best *error reporting* possible
+* *not generating* reams of code for you to debug
+
+Parslet takes the long way around to make *your job* easier. It allows for
+incremental language construction. Often, you start out small, implementing
+the atoms of your language first; _parslet_ takes pride in making this
+possible. 
+
+Eager to try this out? Please see the associated web site:
+http://kschiess.github.com/parslet
 
 SYNOPSIS
 
@@ -56,7 +25,7 @@ SYNOPSIS
               str('"').absnt? >> any
             ).repeat.as(:string) >> 
             str('"')
-    
+  
   # Parse the string and capture parts of the interpretation (:string above)        
   tree = parser.parse(%Q{
     "This is a \\"String\\" in which you can escape stuff"
@@ -64,36 +33,24 @@ SYNOPSIS
 
   tree # => {:string=>"This is a \\\"String\\\" in which you can escape stuff"}
 
-  # Here's how you can grab results from that tree: 
+  # Here's how you can grab results from that tree, two methods: 
+
+  # 1)
   Pattern.new(:string => simple(:x)).each_match(tree) do |dictionary|
-    puts "String contents: #{dictionary[:x]}"
+    puts "String contents (method 1): #{dictionary[:x]}"
   end
-  
-  # Here's how to transform that tree into something else ----------------------
 
-  # Defines the classes of our new Syntax Tree
-  class StringLiteral < Struct.new(:text); end
-
-  # Defines a set of transformation rules on tree leafes
-  transform = Transform.new
-  transform.rule(:string => simple(:x)) { |d| StringLiteral.new(d[:x]) }
-
-  # Transforms the tree
-  transform.apply(tree) 
-
-  # => #<struct StringLiteral text="This is a \\\"String\\\" ... escape stuff">  
+  # 2)
+  transform = Parslet::Transform.new do
+    rule(:string => simple(:x)) { 
+      puts "String contents (method 2): #{x}" }
+  end
+  transform.apply(tree)
 
 COMPATIBILITY
 
 This library should work with both ruby 1.8 and ruby 1.9.
 
-AUTHORS
-
-My gigantous thanks go to the following cool guys and gals that help make this
-rock:
-
-Florian Hanke <florian.hanke@gmail.com>
-
 STATUS 
 
 On the road to 1.0; improving documentation, packaging and upgrading to rspec2.

data/Rakefile

@@ -18,7 +18,7 @@ spec = Gem::Specification.new do |s|
 
   # Change these as appropriate
   s.name              = "parslet"
-  s.version           = "0.9.0"
+  s.version           = "0.10.1"
   s.summary           = "Parser construction library with great error reporting in Ruby."
   s.author            = "Kaspar Schiess"
   s.email             = "kaspar.schiess@absurd.li"
@@ -34,7 +34,7 @@ spec = Gem::Specification.new do |s|
 
   # If you want to depend on other gems, add them here, along with any
   # relevant versions
-  # s.add_dependency("some_other_gem", "~> 0.1.0")
+  s.add_dependency("blankslate", "~> 2.1.2.3")
 
   # If your tests use any gems, include them here
   s.add_development_dependency("rspec")
@@ -60,11 +60,15 @@ end
 
 task :package => :gemspec
 
+require 'sdoc'
+
 # Generate documentation
-Rake::RDocTask.new do |rd|
-  rd.main = "README"
-  rd.rdoc_files.include("README", "lib/**/*.rb")
-  rd.rdoc_dir = "rdoc"
+Rake::RDocTask.new do |rdoc|
+  rdoc.options << '--fmt' << 'shtml' # explictly set shtml generator
+  rdoc.template = 'direct' # lighter template used on railsapi.com
+  rdoc.main = "README"
+  rdoc.rdoc_files.include("README", "lib/**/*.rb")
+  rdoc.rdoc_dir = "rdoc"
 end
 
 desc 'Clear out RDoc and generated packages'

data/lib/parslet.rb

@@ -4,14 +4,9 @@ require 'stringio'
 #
 #   require 'parslet'
 #        
-#   class MyParser
-#     include Parslet
-#        
+#   class MyParser < Parslet::Parser
 #     rule(:a) { str('a').repeat }
-#        
-#     def parse(str)
-#       a.parse(str)
-#     end
+#     root(:a)        
 #   end
 #        
 #   pp MyParser.new.parse('aaaa')   # => 'aaaa'
@@ -36,138 +31,19 @@ require 'stringio'
 # and use the second stage to isolate the rest of your code from the changes
 # you've effected. 
 #
-# = Language Atoms
-#
-# PEG-style grammars build on a very small number of atoms, or parslets. In
-# fact, only three types of parslets exist. Here's how to match a string: 
-#
-#   str('a string')
-#
-# This matches the string 'a string' literally and nothing else. If your input
-# doesn't contain the string, it will fail. Here's how to match a character
-# set: 
-#
-#   match('[abc]')
-#
-# This matches 'a', 'b' or 'c'. The string matched will always have a length
-# of 1; to match longer strings, please see the title below. The last parslet
-# of the three is 'any':
-#
-#   any
-#
-# 'any' functions like the dot in regular expressions - it matches any single
-# character. 
-#
-# = Combination and Repetition
-#
-# Parslets only get useful when combined to grammars. To combine one parslet
-# with the other, you have 4 kinds of methods available: repeat and maybe, >>
-# (sequence), | (alternation), absnt? and prsnt?.
-#
-#   str('a').repeat     # any number of 'a's, including 0
-#   str('a').maybe      # maybe there'll be an 'a', maybe not   
-#
-# Parslets can be joined using >>. This means: Match the left parslet, then
-# match the right parslet. 
-#
-#   str('a') >> str('b')  # would match 'ab'
-#
-# Keep in mind that all combination and repetition operators themselves return
-# a parslet. You can combine the result again: 
-#
-#   ( str('a') >> str('b') ) >> str('c')    # would match 'abc'
-#    
-# The slash ('|') indicates alternatives: 
-#
-#   str('a') | str('b')   # would match 'a' OR 'b'
-#
-# The left side of an alternative is matched first; if it matches, the right
-# side is never looked at. 
-#
-# The absnt? and prsnt? qualifiers allow looking at input without consuming
-# it: 
-#
-#   str('a').absnt?               # will match if at the current position there is an 'a'. 
-#   str('a').absnt? >> str('b')   # check for 'a' then match 'b'
-#
-# This means that the second example will not match any input; when the second
-# part is parsed, the first part has asserted the presence of 'a', and thus
-# str('b') cannot match. The prsnt? method is the opposite of absnt?, it
-# asserts presence. 
-#    
-# More documentation on these methods can be found in Parslets::Atoms::Base.
-#
-# = Intermediary Parse Trees
-# 
-# As you have probably seen above, you can hand input (strings or StringIOs) to
-# your parslets like this: 
-#
-#   parslet.parse(str)    
-#
-# This returns an intermediary parse tree or raises an exception
-# (Parslet::ParseFailed) when the input is not well formed. 
-#
-# Intermediary parse trees are essentially just Plain Old Ruby Objects. (PORO
-# technology as we call it.) Parslets try very hard to return sensible stuff; 
-# it is quite easy to use the results for the later stages of your program. 
-#
-# Here a few examples and what their intermediary tree looks like: 
-#
-#   str('foo').parse('foo')                           # => 'foo'
-#   (str('f') >> str('o') >> str('o')).parse('foo')   # => 'foo'
-#   
-# Naming parslets
-#
-# Construction of lambda blocks
-#
-# = Intermediary Tree transformation
-#
-# The intermediary parse tree by itself is most often not very useful. Its
-# form is volatile; changing your parser in the slightest might produce
-# profound changes in the generated trees. 
-#
-# Generally you will want to construct a more stable tree using your own
-# carefully crafted representation of the domain. Parslet provides you with
-# an elegant way of transmogrifying your intermediary tree into the output
-# format you choose. This is achieved by transformation rules such as this
-# one: 
-#
-#   transform.rule(:literal => {:string => :_x}) { |d| 
-#     StringLit.new(*d.values) }
-# 
-# The above rule will transform a subtree looking like this: 
-#
-#                              :literal
-#                                   |     
-#                               :string    
-#                                   |
-#                              "somestring"
-#
-# into just this: 
-#
-#                               StringLit
-#                               value: "somestring"
-#
-#
-# = Further documentation
-#
-# Please see the examples subdirectory of the distribution for more examples.
-# Check out 'rooc' (github.com/kschiess/rooc) as well - it uses parslet for
-# compiler construction. 
-#       
 module Parslet
   def self.included(base)
     base.extend(ClassMethods)
   end
   
-  # This is raised when the parse failed to match or to consume all its input.
-  # It contains the message that should be presented to the user. If you want
-  # to display more error explanation, you can print the #error_tree that is
+  # Raised when the parse failed to match or to consume all its input. It
+  # contains the message that should be presented to the user. If you want to
+  # display more error explanation, you can print the #error_tree that is
   # stored in the parslet. This is a graphical representation of what went
   # wrong. 
   #
   # Example: 
-  #    
+  #     
   #   begin
   #     parslet.parse(str)
   #   rescue Parslet::ParseFailed => failure
@@ -181,6 +57,7 @@ module Parslet
     # Define the parsers #root function. This is the place where you start 
     # parsing; if you have a rule for 'file' that describes what should be 
     # in a file, this would be your root declaration: 
+    #
     #   class Parser
     #     root :file
     #     rule(:file) { ... }
@@ -205,9 +82,9 @@ module Parslet
       end
     end
     
-    # Define an entity for the parser. This generates a method of the same name
-    # that can be used as part of other patterns. Those methods can be freely
-    # mixed in your parser class with real ruby methods.
+    # Define an entity for the parser. This generates a method of the same
+    # name that can be used as part of other patterns. Those methods can be
+    # freely mixed in your parser class with real ruby methods.
     #
     # Example: 
     #
@@ -233,6 +110,14 @@ module Parslet
     end
   end
   
+  # Allows for delayed construction of #match.
+  #
+  class DelayedMatchConstructor
+    def [](str)
+      Atoms::Re.new("[" + str + "]")
+    end
+  end
+  
   # Returns an atom matching a character class. This is essentially a regular
   # expression, but you should only match a single character. 
   #
@@ -241,8 +126,10 @@ module Parslet
   #   match('[ab]')     # will match either 'a' or 'b'
   #   match('[\n\s]')   # will match newlines and spaces
   #
-  def match(obj)
-    Atoms::Re.new(obj)
+  def match(str=nil)
+    return DelayedMatchConstructor.new unless str
+    
+    return Atoms::Re.new(str)
   end
   module_function :match
   
@@ -263,7 +150,19 @@ module Parslet
     Atoms::Re.new('.')
   end
   module_function :any
-
+  
+  # A special kind of atom that allows embedding whole treetop expressions
+  # into parslet construction. 
+  #
+  # Example: 
+  #
+  #   exp(%Q("a" "b"?))     # => returns the same as str('a') >> str('b').maybe
+  #
+  def exp(str)
+    Parslet::Expression.new(str).to_parslet
+  end
+  module_function :exp
+  
   # Returns a placeholder for a tree transformation that will only match a 
   # sequence of elements. The +symbol+ you specify will be the key for the 
   # matched sequence in the returned dictionary.
@@ -292,10 +191,24 @@ module Parslet
     Pattern::SimpleBind.new(symbol)
   end
   module_function :simple
+  
+  # Returns a placeholder for tree transformation patterns that will match 
+  # any kind of subtree. 
+  #
+  # Example: 
+  #
+  #   { :expression => subtree(:exp) }
+  #
+  def subtree(symbol)
+    Pattern::SubtreeBind.new(symbol)
+  end
+  
+  autoload :Expression, 'parslet/expression'
 end
 
 require 'parslet/error_tree'
 require 'parslet/atoms'
 require 'parslet/pattern'
 require 'parslet/pattern/binding'
-require 'parslet/transform'
+require 'parslet/transform'
+require 'parslet/parser'

data/lib/parslet/atoms.rb

@@ -1,4 +1,7 @@
 module Parslet::Atoms
+  # The precedence module controls parenthesis during the #inspect printing
+  # of parslets. It is not relevant to other aspects of the parsing. 
+  #
   module Precedence
     prec = 0
     BASE       = (prec+=1)    # everything else
@@ -9,484 +12,14 @@ module Parslet::Atoms
     OUTER      = (prec+=1)    # printing is done here.
   end
   
-  # Base class for all parslets, handles orchestration of calls and implements
-  # a lot of the operator and chaining methods.
-  #
-  class Base
-    def parse(io)
-      if io.respond_to? :to_str
-        io = StringIO.new(io)
-      end
-      
-      result = apply(io)
-      
-      # If we haven't consumed the input, then the pattern doesn't match. Try
-      # to provide a good error message (even asking down below)
-      unless io.eof?
-        # Do we know why we stopped matching input? If yes, that's a good
-        # error to fail with. Otherwise just report that we cannot consume the
-        # input.
-        if cause 
-          raise Parslet::ParseFailed, "Unconsumed input, maybe because of this: #{cause}"
-        else
-          error(io, "Don't know what to do with #{io.string[io.pos,100]}") 
-        end
-      end
-      
-      return flatten(result)
-    end
-
-    def apply(io)
-      # p [:start, self, io.string[io.pos, 10]]
-      
-      old_pos = io.pos
-      
-      # p [:try, self, io.string[io.pos, 20]]
-      begin
-        r = try(io)
-        # p [:return_from, self, flatten(r)]
-        @last_cause = nil
-        return r
-      rescue Parslet::ParseFailed => ex
-        # p [:failing, self, io.string[io.pos, 20]]
-        io.pos = old_pos; raise ex
-      end
-    end
-    
-    def repeat(min=0, max=nil)
-      Repetition.new(self, min, max)
-    end
-    def maybe
-      Repetition.new(self, 0, 1, :maybe)
-    end
-    def >>(parslet)
-      Sequence.new(self, parslet)
-    end
-    def |(parslet)
-      Alternative.new(self, parslet)
-    end
-    def absnt?
-      Lookahead.new(self, false)
-    end
-    def prsnt?
-      Lookahead.new(self, true)
-    end
-    def as(name)
-      Named.new(self, name)
-    end
-
-    def flatten(value)
-      # Passes through everything that isn't an array of things
-      return value unless value.instance_of? Array
-
-      # Extracts the s-expression tag
-      tag, *tail = value
-      
-      # Merges arrays:
-      result = tail.
-        map { |e| flatten(e) }            # first flatten each element
-        
-      case tag
-        when :sequence
-          return flatten_sequence(result)
-        when :maybe
-          return result.first
-        when :repetition
-          return flatten_repetition(result)
-      end
-      
-      fail "BUG: Unknown tag #{tag.inspect}."
-    end
-    def flatten_sequence(list)
-      list.inject('') { |r, e|        # and then merge flat elements
-        case [r, e].map { |o| o.class }
-          when [Hash, Hash]             # two keyed subtrees: make one
-            warn_about_duplicate_keys(r, e)
-            r.merge(e)
-          # a keyed tree and an array (push down)
-          when [Hash, Array]
-            [r] + e
-          when [Array, Hash]   
-            r + [e]
-          when [String, String]
-            r << e
-        else
-          if r.instance_of? Hash
-            r   # Ignore e, since its not a hash we can merge
-          else
-            e   # Whatever e is at this point, we keep it
-          end
-        end
-      }
-    end
-    def flatten_repetition(list)
-      if list.any? { |e| e.instance_of?(Hash) }
-        # If keyed subtrees are in the array, we'll want to discard all 
-        # strings inbetween. To keep them, name them. 
-        return list.select { |e| e.instance_of?(Hash) }
-      end
-
-      if list.any? { |e| e.instance_of?(Array) }
-        # If any arrays are nested in this array, flatten all arrays to this
-        # level. 
-        return list.
-          select { |e| e.instance_of?(Array) }.
-          flatten(1)
-      end
-      
-      # If there are only strings, concatenate them and return that. 
-      list.inject('') { |s,e| s<<(e||'') }
-    end
-
-    def self.precedence(prec)
-      define_method(:precedence) { prec }
-    end
-    precedence Precedence::BASE
-    def to_s(outer_prec)
-      if outer_prec < precedence
-        "("+to_s_inner(precedence)+")"
-      else
-        to_s_inner(precedence)
-      end
-    end
-    def inspect
-      to_s(Precedence::OUTER)
-    end
-
-    # Cause should return the current best approximation of this parslet
-    # of what went wrong with the parse. Not relevant if the parse succeeds, 
-    # but needed for clever error reports. 
-    #
-    def cause
-      @last_cause
-    end
-
-    # Error tree returns what went wrong here plus what went wrong inside 
-    # subexpressions as a tree. The error stored for this node will be equal
-    # with #cause. 
-    #
-    def error_tree
-      Parslet::ErrorTree.new(self) if cause?
-    end
-    def cause?
-      not @last_cause.nil?
-    end
-  private
-    # Report/raise a parse error with the given message, printing the current
-    # position as well. Appends 'at line X char Y.' to the message you give. 
-    # If +pos+ is given, it is used as the real position the error happened, 
-    # correcting the io's current position.
-    #
-    def error(io, str, pos=nil)
-      pre = io.string[0..(pos||io.pos)]
-      lines = Array(pre.lines)
-      
-      if lines.empty?
-        formatted_cause = str
-      else
-        pos   = lines.last.length
-        formatted_cause = "#{str} at line #{lines.count} char #{pos}."
-      end
-
-      @last_cause = formatted_cause
-      
-      raise Parslet::ParseFailed, formatted_cause, nil
-    end
-    def warn_about_duplicate_keys(h1, h2)
-      d = h1.keys & h2.keys
-      unless d.empty?
-        warn "Duplicate subtrees while merging result of \n  #{self.inspect}\nonly the values"+
-             " of the latter will be kept. (keys: #{d.inspect})"
-      end
-    end
-  end
-  
-  class Named < Base
-    attr_reader :parslet, :name
-    def initialize(parslet, name)
-      @parslet, @name = parslet, name
-    end
-    
-    def apply(io)
-      value = parslet.apply(io)
-      
-      produce_return_value value
-    end
-    
-    def to_s_inner(prec)
-      "#{name}:#{parslet.to_s(prec)}"
-    end
-
-    def error_tree
-      parslet.error_tree
-    end
-  private
-    def produce_return_value(val)  
-      { name => flatten(val) }
-    end
-  end
-  
-  class Lookahead < Base
-    attr_reader :positive
-    attr_reader :bound_parslet
-    
-    def initialize(bound_parslet, positive=true)
-      # Model positive and negative lookahead by testing this flag.
-      @positive = positive
-      @bound_parslet = bound_parslet
-    end
-    
-    def try(io)
-      pos = io.pos
-      begin
-        bound_parslet.apply(io)
-      rescue Parslet::ParseFailed 
-        return fail(io)
-      ensure 
-        io.pos = pos
-      end
-      return success(io)
-    end
-    
-    def fail(io)
-      if positive
-        error(io, "lookahead: #{bound_parslet.inspect} didn't match, but should have")
-      else
-        # TODO: Squash this down to nothing? Return value handling here...
-        return nil
-      end
-    end
-    def success(io)
-      if positive
-        return nil  # see above, TODO
-      else
-        error(
-          io, 
-          "negative lookahead: #{bound_parslet.inspect} matched, but shouldn't have")
-      end
-    end
-
-    precedence Precedence::LOOKAHEAD
-    def to_s_inner(prec)
-      char = positive ? '&' : '!'
-      
-      "#{char}#{bound_parslet.to_s(prec)}"
-    end
-
-    def error_tree
-      bound_parslet.error_tree
-    end
-  end
-
-  class Alternative < Base
-    attr_reader :alternatives
-    def initialize(*alternatives)
-      @alternatives = alternatives
-    end
-    
-    def |(parslet)
-      @alternatives << parslet
-      self
-    end
-    
-    def try(io)
-      alternatives.each { |a|
-        begin
-          return a.apply(io)
-        rescue Parslet::ParseFailed => ex
-        end
-      }
-      # If we reach this point, all alternatives have failed. 
-      error(io, "Expected one of #{alternatives.inspect}.")
-    end
-
-    precedence Precedence::ALTERNATE
-    def to_s_inner(prec)
-      alternatives.map { |a| a.to_s(prec) }.join(' | ')
-    end
-
-    def error_tree
-      Parslet::ErrorTree.new(self, *alternatives.
-        map { |child| child.error_tree })
-    end
-  end
-  
-  # A sequence of parslets, matched from left to right. Denoted by '>>'
-  #
-  class Sequence < Base
-    attr_reader :parslets
-    def initialize(*parslets)
-      @parslets = parslets
-    end
-    
-    def >>(parslet)
-      @parslets << parslet
-      self
-    end
-    
-    def try(io)
-      [:sequence]+parslets.map { |p| 
-        # Save each parslet as potentially offending (raising an error). 
-        @offending_parslet = p
-        p.apply(io) 
-      }
-    rescue Parslet::ParseFailed
-      error(io, "Failed to match sequence (#{self.inspect})")
-    end
-        
-    precedence Precedence::SEQUENCE
-    def to_s_inner(prec)
-      parslets.map { |p| p.to_s(prec) }.join(' ')
-    end
-
-    def error_tree
-      Parslet::ErrorTree.new(self).tap { |t|
-        t.children << @offending_parslet.error_tree if @offending_parslet }
-    end
-  end
-  
-  class Repetition < Base
-    attr_reader :min, :max, :parslet
-    def initialize(parslet, min, max, tag=:repetition)
-      @parslet = parslet
-      @min, @max = min, max
-      @tag = tag
-    end
-    
-    def try(io)
-      occ = 0
-      result = [@tag]   # initialize the result array with the tag (for flattening)
-      loop do
-        begin
-          result << parslet.apply(io)
-          occ += 1
-
-          # If we're not greedy (max is defined), check if that has been 
-          # reached. 
-          return result if max && occ>=max
-        rescue Parslet::ParseFailed => ex
-          # Greedy matcher has produced a failure. Check if occ (which will
-          # contain the number of sucesses) is in {min, max}.
-          # p [:repetition, occ, min, max]
-          error(io, "Expected at least #{min} of #{parslet.inspect}") if occ < min
-          return result
-        end
-      end
-    end
-    
-    precedence Precedence::REPETITION
-    def to_s_inner(prec)
-      minmax = "{#{min}, #{max}}"
-      minmax = '?' if min == 0 && max == 1
-
-      parslet.to_s(prec) + minmax
-    end
-
-    def cause
-      # Either the repetition failed or the parslet inside failed to repeat. 
-      super || parslet.cause
-    end
-    def error_tree
-      if cause?
-        Parslet::ErrorTree.new(self, parslet.error_tree)
-      else
-        parslet.error_tree
-      end
-    end
-  end
-
-  # Matches a special kind of regular expression that only ever matches one
-  # character at a time. Useful members of this family are: character ranges, 
-  # \w, \d, \r, \n, ...
-  #
-  class Re < Base
-    attr_reader :match
-    def initialize(match)
-      @match = match
-    end
-
-    def try(io)
-      r = Regexp.new(match, Regexp::MULTILINE)
-      s = io.read(1)
-      error(io, "Premature end of input") unless s
-      error(io, "Failed to match #{match.inspect[1..-2]}") unless s.match(r)
-      return s
-    end
-
-    def to_s_inner(prec)
-      match.inspect[1..-2]
-    end
-  end
-  
-  # Matches a string of characters. 
-  #
-  class Str < Base
-    attr_reader :str
-    def initialize(str)
-      @str = str
-    end
-    
-    def try(io)
-      old_pos = io.pos
-      s = io.read(str.size)
-      error(io, "Premature end of input") unless s && s.size==str.size
-      error(io, "Expected #{str.inspect}, but got #{s.inspect}", old_pos) \
-        unless s==str
-      return s
-    end
-    
-    def to_s_inner(prec)
-      "'#{str}'"
-    end
-  end
-
-  # This wraps pieces of parslet definition and gives them a name. The wrapped
-  # piece is lazily evaluated and cached. This has two purposes: 
-  #     
-  # a) Avoid infinite recursion during evaluation of the definition
-  #
-  # b) Be able to print things by their name, not by their sometimes
-  #    complicated content.
-  #
-  # You don't normally use this directly, instead you should generated it by
-  # using the structuring method Parslet#rule.
-  #
-  class Entity < Base
-    attr_reader :name, :context, :block
-    def initialize(name, context, block)
-      super()
-      
-      @name = name
-      @context = context
-      @block = block
-    end
-
-    def try(io)
-      parslet.apply(io)
-    end
-    
-    def parslet
-      @parslet ||= context.instance_eval(&block).tap { |p| 
-        raise_not_implemented unless p
-      }
-    end
-
-    def to_s_inner(prec)
-      name.to_s.upcase
-    end
-
-    def error_tree
-      parslet.error_tree
-    end
-    
-  private 
-    def raise_not_implemented
-      trace = caller.reject {|l| l =~ %r{#{Regexp.escape(__FILE__)}}} # blatantly stolen from dependencies.rb in activesupport
-      exception = NotImplementedError.new("rule(#{name.inspect}) { ... }  returns nil. Still not implemented, but already used?")
-      exception.set_backtrace(trace)
-      
-      raise exception
-    end
-  end
+  autoload :Base,         'parslet/atoms/base'
+  autoload :Named,        'parslet/atoms/named'
+  autoload :Lookahead,    'parslet/atoms/lookahead'
+  autoload :Alternative,  'parslet/atoms/alternative'
+  autoload :Sequence,     'parslet/atoms/sequence'
+  autoload :Repetition,   'parslet/atoms/repetition'
+  autoload :Re,           'parslet/atoms/re'
+  autoload :Str,          'parslet/atoms/str'
+  autoload :Entity,       'parslet/atoms/entity'
 end