mail 2.1.5.2 → 2.1.5.3

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of mail might be problematic. Click here for more details.

Files changed (134) hide show
  1. data/CHANGELOG.rdoc +6 -0
  2. data/Rakefile +2 -1
  3. data/lib/mail.rb +0 -1
  4. data/lib/mail/elements.rb +0 -1
  5. data/lib/mail/utilities.rb +1 -1
  6. data/lib/mail/version.rb +1 -1
  7. data/lib/tasks/treetop.rake +4 -4
  8. metadata +17 -129
  9. data/lib/mail/vendor/treetop-1.4.4/History.txt +0 -18
  10. data/lib/mail/vendor/treetop-1.4.4/LICENSE +0 -19
  11. data/lib/mail/vendor/treetop-1.4.4/README.md +0 -164
  12. data/lib/mail/vendor/treetop-1.4.4/Rakefile +0 -28
  13. data/lib/mail/vendor/treetop-1.4.4/Treetop.tmbundle/Preferences/Comments.tmPreferences +0 -22
  14. data/lib/mail/vendor/treetop-1.4.4/Treetop.tmbundle/Snippets/grammar ___ end.tmSnippet +0 -20
  15. data/lib/mail/vendor/treetop-1.4.4/Treetop.tmbundle/Snippets/rule ___ end.tmSnippet +0 -18
  16. data/lib/mail/vendor/treetop-1.4.4/Treetop.tmbundle/Support/nibs/SyntaxTreeViewer.nib/designable.nib +0 -1524
  17. data/lib/mail/vendor/treetop-1.4.4/Treetop.tmbundle/Support/nibs/SyntaxTreeViewer.nib/keyedobjects.nib +0 -0
  18. data/lib/mail/vendor/treetop-1.4.4/Treetop.tmbundle/Support/syntax_tree_viewer.rb +0 -117
  19. data/lib/mail/vendor/treetop-1.4.4/Treetop.tmbundle/Syntaxes/Treetop Grammar.tmLanguage +0 -278
  20. data/lib/mail/vendor/treetop-1.4.4/Treetop.tmbundle/info.plist +0 -10
  21. data/lib/mail/vendor/treetop-1.4.4/benchmark/seqpar.gnuplot +0 -15
  22. data/lib/mail/vendor/treetop-1.4.4/benchmark/seqpar.treetop +0 -16
  23. data/lib/mail/vendor/treetop-1.4.4/benchmark/seqpar_benchmark.rb +0 -107
  24. data/lib/mail/vendor/treetop-1.4.4/bin/tt +0 -112
  25. data/lib/mail/vendor/treetop-1.4.4/doc/contributing_and_planned_features.markdown +0 -103
  26. data/lib/mail/vendor/treetop-1.4.4/doc/grammar_composition.markdown +0 -65
  27. data/lib/mail/vendor/treetop-1.4.4/doc/index.markdown +0 -90
  28. data/lib/mail/vendor/treetop-1.4.4/doc/pitfalls_and_advanced_techniques.markdown +0 -51
  29. data/lib/mail/vendor/treetop-1.4.4/doc/semantic_interpretation.markdown +0 -189
  30. data/lib/mail/vendor/treetop-1.4.4/doc/site.rb +0 -112
  31. data/lib/mail/vendor/treetop-1.4.4/doc/sitegen.rb +0 -65
  32. data/lib/mail/vendor/treetop-1.4.4/doc/syntactic_recognition.markdown +0 -100
  33. data/lib/mail/vendor/treetop-1.4.4/doc/using_in_ruby.markdown +0 -28
  34. data/lib/mail/vendor/treetop-1.4.4/examples/lambda_calculus/arithmetic.rb +0 -551
  35. data/lib/mail/vendor/treetop-1.4.4/examples/lambda_calculus/arithmetic.treetop +0 -97
  36. data/lib/mail/vendor/treetop-1.4.4/examples/lambda_calculus/arithmetic_node_classes.rb +0 -7
  37. data/lib/mail/vendor/treetop-1.4.4/examples/lambda_calculus/arithmetic_test.rb +0 -54
  38. data/lib/mail/vendor/treetop-1.4.4/examples/lambda_calculus/lambda_calculus +0 -0
  39. data/lib/mail/vendor/treetop-1.4.4/examples/lambda_calculus/lambda_calculus.rb +0 -718
  40. data/lib/mail/vendor/treetop-1.4.4/examples/lambda_calculus/lambda_calculus.treetop +0 -132
  41. data/lib/mail/vendor/treetop-1.4.4/examples/lambda_calculus/lambda_calculus_node_classes.rb +0 -5
  42. data/lib/mail/vendor/treetop-1.4.4/examples/lambda_calculus/lambda_calculus_test.rb +0 -89
  43. data/lib/mail/vendor/treetop-1.4.4/examples/lambda_calculus/test_helper.rb +0 -18
  44. data/lib/mail/vendor/treetop-1.4.4/lib/treetop.rb +0 -3
  45. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/bootstrap_gen_1_metagrammar.rb +0 -42
  46. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler.rb +0 -7
  47. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/grammar_compiler.rb +0 -44
  48. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/lexical_address_space.rb +0 -17
  49. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/metagrammar.rb +0 -3366
  50. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/metagrammar.treetop +0 -436
  51. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes.rb +0 -19
  52. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes/anything_symbol.rb +0 -18
  53. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes/atomic_expression.rb +0 -14
  54. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes/character_class.rb +0 -28
  55. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes/choice.rb +0 -31
  56. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes/declaration_sequence.rb +0 -24
  57. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes/grammar.rb +0 -28
  58. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes/inline_module.rb +0 -27
  59. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes/nonterminal.rb +0 -13
  60. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes/optional.rb +0 -19
  61. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes/parenthesized_expression.rb +0 -9
  62. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes/parsing_expression.rb +0 -146
  63. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes/parsing_rule.rb +0 -58
  64. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes/predicate.rb +0 -45
  65. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes/predicate_block.rb +0 -16
  66. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes/repetition.rb +0 -55
  67. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes/sequence.rb +0 -71
  68. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes/terminal.rb +0 -20
  69. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes/transient_prefix.rb +0 -9
  70. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/node_classes/treetop_file.rb +0 -9
  71. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/compiler/ruby_builder.rb +0 -113
  72. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/polyglot.rb +0 -9
  73. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/ruby_extensions.rb +0 -1
  74. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/ruby_extensions/string.rb +0 -42
  75. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/runtime.rb +0 -6
  76. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/runtime/compiled_parser.rb +0 -110
  77. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/runtime/interval_skip_list.rb +0 -3
  78. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/runtime/interval_skip_list/head_node.rb +0 -15
  79. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/runtime/interval_skip_list/interval_skip_list.rb +0 -200
  80. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/runtime/interval_skip_list/node.rb +0 -164
  81. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/runtime/syntax_node.rb +0 -114
  82. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/runtime/terminal_parse_failure.rb +0 -16
  83. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/runtime/terminal_syntax_node.rb +0 -17
  84. data/lib/mail/vendor/treetop-1.4.4/lib/treetop/version.rb +0 -9
  85. data/lib/mail/vendor/treetop-1.4.4/script/generate_metagrammar.rb +0 -13
  86. data/lib/mail/vendor/treetop-1.4.4/script/svnadd +0 -11
  87. data/lib/mail/vendor/treetop-1.4.4/script/svnrm +0 -11
  88. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/and_predicate_spec.rb +0 -36
  89. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/anything_symbol_spec.rb +0 -44
  90. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/character_class_spec.rb +0 -276
  91. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/choice_spec.rb +0 -80
  92. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/circular_compilation_spec.rb +0 -30
  93. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/failure_propagation_functional_spec.rb +0 -21
  94. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/grammar_compiler_spec.rb +0 -84
  95. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/grammar_spec.rb +0 -41
  96. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/multibyte_chars_spec.rb +0 -38
  97. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/nonterminal_symbol_spec.rb +0 -40
  98. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/not_predicate_spec.rb +0 -38
  99. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/one_or_more_spec.rb +0 -35
  100. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/optional_spec.rb +0 -37
  101. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/parenthesized_expression_spec.rb +0 -19
  102. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/parsing_rule_spec.rb +0 -61
  103. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/repeated_subrule_spec.rb +0 -29
  104. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/semantic_predicate_spec.rb +0 -175
  105. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/sequence_spec.rb +0 -115
  106. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/terminal_spec.rb +0 -81
  107. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/terminal_symbol_spec.rb +0 -37
  108. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/test_grammar.treetop +0 -7
  109. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/test_grammar.tt +0 -7
  110. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/test_grammar_do.treetop +0 -7
  111. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/tt_compiler_spec.rb +0 -216
  112. data/lib/mail/vendor/treetop-1.4.4/spec/compiler/zero_or_more_spec.rb +0 -56
  113. data/lib/mail/vendor/treetop-1.4.4/spec/composition/a.treetop +0 -11
  114. data/lib/mail/vendor/treetop-1.4.4/spec/composition/b.treetop +0 -11
  115. data/lib/mail/vendor/treetop-1.4.4/spec/composition/c.treetop +0 -10
  116. data/lib/mail/vendor/treetop-1.4.4/spec/composition/d.treetop +0 -10
  117. data/lib/mail/vendor/treetop-1.4.4/spec/composition/f.treetop +0 -17
  118. data/lib/mail/vendor/treetop-1.4.4/spec/composition/grammar_composition_spec.rb +0 -40
  119. data/lib/mail/vendor/treetop-1.4.4/spec/composition/subfolder/e_includes_c.treetop +0 -15
  120. data/lib/mail/vendor/treetop-1.4.4/spec/ruby_extensions/string_spec.rb +0 -32
  121. data/lib/mail/vendor/treetop-1.4.4/spec/runtime/compiled_parser_spec.rb +0 -101
  122. data/lib/mail/vendor/treetop-1.4.4/spec/runtime/interval_skip_list/delete_spec.rb +0 -147
  123. data/lib/mail/vendor/treetop-1.4.4/spec/runtime/interval_skip_list/expire_range_spec.rb +0 -349
  124. data/lib/mail/vendor/treetop-1.4.4/spec/runtime/interval_skip_list/insert_and_delete_node.rb +0 -385
  125. data/lib/mail/vendor/treetop-1.4.4/spec/runtime/interval_skip_list/insert_spec.rb +0 -660
  126. data/lib/mail/vendor/treetop-1.4.4/spec/runtime/interval_skip_list/interval_skip_list_spec.graffle +0 -6175
  127. data/lib/mail/vendor/treetop-1.4.4/spec/runtime/interval_skip_list/interval_skip_list_spec.rb +0 -58
  128. data/lib/mail/vendor/treetop-1.4.4/spec/runtime/interval_skip_list/palindromic_fixture.rb +0 -23
  129. data/lib/mail/vendor/treetop-1.4.4/spec/runtime/interval_skip_list/palindromic_fixture_spec.rb +0 -163
  130. data/lib/mail/vendor/treetop-1.4.4/spec/runtime/interval_skip_list/spec_helper.rb +0 -84
  131. data/lib/mail/vendor/treetop-1.4.4/spec/runtime/syntax_node_spec.rb +0 -77
  132. data/lib/mail/vendor/treetop-1.4.4/spec/spec_helper.rb +0 -110
  133. data/lib/mail/vendor/treetop-1.4.4/treetop.gemspec +0 -18
  134. data/lib/mail/vendor/treetop.rb +0 -2
@@ -1,112 +0,0 @@
1
- #!/usr/bin/env ruby
2
- require 'optparse'
3
- require 'rubygems'
4
-
5
- $LOAD_PATH.unshift(File.expand_path(File.dirname(__FILE__) + "/../lib"))
6
- require 'treetop'
7
- require 'treetop/version'
8
- require 'treetop/polyglot'
9
-
10
- options = {}
11
- parser = OptionParser.new do |opts|
12
- exts = Treetop::Polyglot::VALID_GRAMMAR_EXT.collect { |i| '.' + i }
13
-
14
- opts.banner = "Treetop Parsing Expression Grammar (PEG) Comand Line Compiler"
15
- opts.define_head "Usage: tt [options] grammar_file[#{exts.join('|')}] ..."
16
- opts.separator ''
17
- opts.separator 'Examples:'
18
- opts.separator ' tt foo.tt # 1 grammar -> 1 parser source'
19
- opts.separator ' tt foo bar.treetop # 2 grammars -> 2 separate parsers'
20
- opts.separator ' tt -o alt_name.rb foo # alternately named output file'
21
- opts.separator ''
22
- opts.separator ''
23
- opts.separator 'NOTE: while treetop grammar files *must* have one of the following'
24
- opts.separator 'filename extensions, the extension name is not required when calling'
25
- opts.separator 'the compiler with grammar file names.'
26
- opts.separator ''
27
- opts.separator " Valid extensions: #{exts.join(', ')}"
28
- opts.separator ''
29
- opts.separator ''
30
- opts.separator 'Options:'
31
-
32
- opts.on('-o', '--output FILENAME', 'Write parser source to FILENAME') do |fn|
33
- options[:out_file] = fn
34
- end
35
-
36
- opts.on('-f', '--force', 'Overwrite existing output file(s)') do
37
- options[:force] = true
38
- end
39
-
40
- opts.on_tail('-v', '--version', 'Show Treetop version') do
41
- puts "Treetop v#{Treetop::VERSION::STRING}"
42
- exit
43
- end
44
-
45
- opts.on_tail('-h', '--help', 'Show this help message') do
46
- puts opts
47
- exit
48
- end
49
-
50
- end
51
- file_list = parser.parse!
52
-
53
- # check options and arg constraints
54
- if file_list.empty? || (options[:out_file] && file_list.size > 1)
55
- puts parser
56
- exit 1
57
- end
58
-
59
- def grammar_exist?(filename)
60
- if File.extname(filename).empty?
61
- Treetop::Polyglot::VALID_GRAMMAR_EXT.each do |ext|
62
- fn_ext = "#{filename}.#{ext}"
63
- return true if File.exist?(fn_ext) && !File.zero?(fn_ext)
64
- end
65
- end
66
- File.exist?(filename) && !File.zero?(filename)
67
- end
68
-
69
- def full_grammar_filename(filename)
70
- return filename if !File.extname(filename).empty?
71
- Treetop::Polyglot::VALID_GRAMMAR_EXT.each do |ext|
72
- fn_ext = "#{filename}.#{ext}"
73
- return fn_ext if File.exist?(fn_ext) && !File.zero?(fn_ext)
74
- end
75
- end
76
-
77
- def protect_output?(filename, forced=false)
78
- if !forced and
79
- File.exist?(filename) and
80
- (l=File.open(filename) { |f| f.gets rescue "" }) != Treetop::Compiler::AUTOGENERATED
81
- puts "ERROR: '#{filename}' output already exists; skipping compilation...\n"
82
- return true
83
- end
84
- false
85
- end
86
-
87
- compiler = Treetop::Compiler::GrammarCompiler.new
88
-
89
- while !file_list.empty?
90
- treetop_file = file_list.shift
91
-
92
- # handle nonexistent and existent grammar files mixed together
93
- if !grammar_exist?(treetop_file)
94
- puts "ERROR: input grammar file '#{treetop_file}' does not exist; continuing...\n"
95
- next
96
- end
97
-
98
- # try to compile
99
- treetop_file = full_grammar_filename(treetop_file)
100
- std_output_file = treetop_file.gsub(Treetop::Polyglot::VALID_GRAMMAR_EXT_REGEXP, '.rb')
101
-
102
- if options[:out_file]
103
- # explicit output file name option; never overwrite unless forced
104
- next if protect_output?(options[:out_file], options[:force])
105
- compiler.compile(treetop_file, options[:out_file])
106
- else
107
- # compile one input file from input file list option; never overwrite unless forced
108
- next if protect_output?(std_output_file, options[:force])
109
- compiler.compile(treetop_file)
110
- end
111
-
112
- end
@@ -1,103 +0,0 @@
1
- #Google Group
2
- I've created a <a href="http://groups.google.com/group/treetop-dev">Google Group</a> as a better place to organize discussion and development.
3
- treetop-dev@google-groups.com
4
-
5
- #Contributing
6
- Visit <a href="http://github.com/nathansobo/treetop/tree/master">the Treetop repository page on GitHub</a> in your browser for more information about checking out the source code.
7
-
8
- I like to try Rubinius's policy regarding commit rights. If you submit one patch worth integrating, I'll give you commit rights. We'll see how this goes, but I think it's a good policy.
9
-
10
-
11
- ##Getting Started with the Code
12
- Treetop compiler is interesting in that it is implemented in itself. Its functionality revolves around `metagrammar.treetop`, which specifies the grammar for Treetop grammars. I took a hybrid approach with regard to definition of methods on syntax nodes in the metagrammar. Methods that are more syntactic in nature, like those that provide access to elements of the syntax tree, are often defined inline, directly in the grammar. More semantic methods are defined in custom node classes.
13
-
14
- Iterating on the metagrammar is tricky. The current testing strategy uses the last stable version of Treetop to parse the version under test. Then the version under test is used to parse and functionally test the various pieces of syntax it should recognize and translate to Ruby. As you change `metagrammar.treetop` and its associated node classes, note that the node classes you are changing are also used to support the previous stable version of the metagrammar, so must be kept backward compatible until such time as a new stable version can be produced to replace it.
15
-
16
- ##Tests
17
- Most of the compiler's tests are functional in nature. The grammar under test is used to parse and compile piece of sample code. Then I attempt to parse input with the compiled output and test its results.
18
-
19
- #What Needs to be Done
20
- ##Small Stuff
21
- * Improve the `tt` command line tool to allow `.treetop` extensions to be elided in its arguments.
22
- * Generate and load temp files with `Treetop.load` rather than evaluating strings to improve stack trace readability.
23
- * Allow `do/end` style blocks as well as curly brace blocks. This was originally omitted because I thought it would be confusing. It probably isn't.
24
-
25
- ##Big Stuff
26
- ####Transient Expressions
27
- Currently, every parsing expression instantiates a syntax node. This includes even very simple parsing expressions, like single characters. It is probably unnecessary for every single expression in the parse to correspond to its own syntax node, so much savings could be garnered from a transient declaration that instructs the parser only to attempt a match without instantiating nodes.
28
-
29
- ###Generate Rule Implementations in C
30
- Parsing expressions are currently compiled into simple Ruby source code that comprises the body of parsing rules, which are translated into Ruby methods. The generator could produce C instead of Ruby in the body of these method implementations.
31
-
32
- ###Global Parsing State and Semantic Backtrack Triggering
33
- Some programming language grammars are not entirely context-free, requiring that global state dictate the behavior of the parser in certain circumstances. Treetop does not currently expose explicit parser control to the grammar writer, and instead automatically constructs the syntax tree for them. A means of semantic parser control compatible with this approach would involve callback methods defined on parsing nodes. Each time a node is successfully parsed it will be given an opportunity to set global state and optionally trigger a parse failure on _extrasyntactic_ grounds. Nodes will probably need to define an additional method that undoes their changes to global state when there is a parse failure and they are backtracked.
34
-
35
- Here is a sketch of the potential utility of such mechanisms. Consider the structure of YAML, which uses indentation to indicate block structure.
36
-
37
- level_1:
38
- level_2a:
39
- level_2b:
40
- level_3a:
41
- level_2c:
42
-
43
- Imagine a grammar like the following:
44
-
45
- rule yaml_element
46
- name ':' block
47
- /
48
- name ':' value
49
- end
50
-
51
- rule block
52
- indent yaml_elements outdent
53
- end
54
-
55
- rule yaml_elements
56
- yaml_element (samedent yaml_element)*
57
- end
58
-
59
- rule samedent
60
- newline spaces {
61
- def after_success(parser_state)
62
- spaces.length == parser_state.indent_level
63
- end
64
- }
65
- end
66
-
67
- rule indent
68
- newline spaces {
69
- def after_success(parser_state)
70
- if spaces.length == parser_state.indent_level + 2
71
- parser_state.indent_level += 2
72
- true
73
- else
74
- false # fail the parse on extrasyntactic grounds
75
- end
76
- end
77
-
78
- def undo_success(parser_state)
79
- parser_state.indent_level -= 2
80
- end
81
- }
82
- end
83
-
84
- rule outdent
85
- newline spaces {
86
- def after_success(parser_state)
87
- if spaces.length == parser_state.indent_level - 2
88
- parser_state.indent_level -= 2
89
- true
90
- else
91
- false # fail the parse on extrasyntactic grounds
92
- end
93
- end
94
-
95
- def undo_success(parser_state)
96
- parser_state.indent_level += 2
97
- end
98
- }
99
- end
100
-
101
- In this case a block will be detected only if a change in indentation warrants it. Note that this change in the state of indentation must be undone if a subsequent failure causes this node not to ultimately be incorporated into a successful result.
102
-
103
- I am by no means sure that the above sketch is free of problems, or even that this overall strategy is sound, but it seems like a promising path.
@@ -1,65 +0,0 @@
1
- #Grammar Composition
2
- A unique property of parsing expression grammars is that they are _closed under composition_. This means that when you compose two grammars they yield another grammar that can be composed yet again. This is a radical departure from parsing frameworks require on lexical scanning, which makes compositionally impossible. Treetop's facilities for composition are built upon those of Ruby.
3
-
4
- ##The Mapping of Treetop Constructs to Ruby Constructs
5
- When Treetop compiles a grammar definition, it produces a module and a class. The module contains methods implementing all of the rules defined in the grammar. The generated class is a subclass of Treetop::Runtime::CompiledParser and includes the module. For example:
6
-
7
- grammar Foo
8
- ...
9
- end
10
-
11
- results in a Ruby module named `Foo` and a Ruby class named `FooParser` that `include`s the `Foo` module.
12
-
13
- ##Using Mixin Semantics to Compose Grammars
14
- Because grammars are just modules, they can be mixed into one another. This enables grammars to share rules.
15
-
16
- grammar A
17
- rule a
18
- 'a'
19
- end
20
- end
21
-
22
- grammar B
23
- include A
24
-
25
- rule ab
26
- a 'b'
27
- end
28
- end
29
-
30
- Grammar `B` above references rule `a` defined in a separate grammar that it includes. Because module inclusion places modules in the ancestor chain, rules may also be overridden with the use of the `super` keyword accessing the overridden rule.
31
-
32
- grammar A
33
- rule a
34
- 'a'
35
- end
36
- end
37
-
38
- grammar B
39
- include A
40
-
41
- rule a
42
- super / 'b'
43
- end
44
- end
45
-
46
- Now rule `a` in grammar `B` matches either `'a'` or `'b'`.
47
-
48
- ##Motivation
49
- Imagine a grammar for Ruby that took account of SQL queries embedded in strings within the language. That could be achieved by combining two existing grammars.
50
-
51
- grammar RubyPlusSQL
52
- include Ruby
53
- include SQL
54
-
55
- rule expression
56
- ruby_expression
57
- end
58
-
59
- rule ruby_string
60
- ruby_quote sql_expression ruby_quote / ruby_string
61
- end
62
- end
63
-
64
- ##Work to be Done
65
- It has become clear that the include facility in grammars would be more useful if it had the ability to name prefix all rules from the included grammar to avoid collision. This is a planned but currently unimplemented feature.
@@ -1,90 +0,0 @@
1
- <p class="intro_text">
2
-
3
- Treetop is a language for describing languages. Combining the elegance of Ruby with cutting-edge <em>parsing expression grammars</em>, it helps you analyze syntax with revolutionarily ease.
4
-
5
- </p>
6
-
7
- sudo gem install treetop
8
-
9
- #Intuitive Grammar Specifications
10
- Parsing expression grammars (PEGs) are simple to write and easy to maintain. They are a simple but powerful generalization of regular expressions that are easier to work with than the LALR or LR-1 grammars of traditional parser generators. There's no need for a tokenization phase, and _lookahead assertions_ can be used for a limited degree of context-sensitivity. Here's an extremely simple Treetop grammar that matches a subset of arithmetic, respecting operator precedence:
11
-
12
- grammar Arithmetic
13
- rule additive
14
- multitive '+' additive / multitive
15
- end
16
-
17
- rule multitive
18
- primary '*' multitive / primary
19
- end
20
-
21
- rule primary
22
- '(' additive ')' / number
23
- end
24
-
25
- rule number
26
- [1-9] [0-9]*
27
- end
28
- end
29
-
30
-
31
- #Syntax-Oriented Programming
32
- Rather than implementing semantic actions that construct parse trees, Treetop lets you define methods on trees that it constructs for you automatically. You can define these methods directly within the grammar...
33
-
34
- grammar Arithmetic
35
- rule additive
36
- multitive '+' additive {
37
- def value
38
- multitive.value + additive.value
39
- end
40
- }
41
- /
42
- multitive
43
- end
44
-
45
- # other rules below ...
46
- end
47
-
48
- ...or associate rules with classes of nodes you wish your parsers to instantiate upon matching a rule.
49
-
50
- grammar Arithmetic
51
- rule additive
52
- multitive '+' additive <AdditiveNode>
53
- /
54
- multitive
55
- end
56
-
57
- # other rules below ...
58
- end
59
-
60
-
61
- #Reusable, Composable Language Descriptions
62
- Because PEGs are closed under composition, Treetop grammars can be treated like Ruby modules. You can mix them into one another and override rules with access to the `super` keyword. You can break large grammars down into coherent units or make your language's syntax modular. This is especially useful if you want other programmers to be able to reuse your work.
63
-
64
- grammar RubyWithEmbeddedSQL
65
- include SQL
66
-
67
- rule string
68
- quote sql_expression quote / super
69
- end
70
- end
71
-
72
-
73
- #Acknowledgements
74
-
75
-
76
- <a href="http://pivotallabs.com"><img id="pivotal_logo" src="./images/pivotal.gif"></a>
77
-
78
- First, thank you to my employer Rob Mee of <a href="http://pivotallabs.com"/>Pivotal Labs</a> for funding a substantial portion of Treetop's development. He gets it.
79
-
80
-
81
- I'd also like to thank:
82
-
83
- * Damon McCormick for several hours of pair programming.
84
- * Nick Kallen for lots of well-considered feedback and a few afternoons of programming.
85
- * Brian Takita for a night of pair programming.
86
- * Eliot Miranda for urging me rewrite as a compiler right away rather than putting it off.
87
- * Ryan Davis and Eric Hodel for hurting my code.
88
- * Dav Yaginuma for kicking me into action on my idea.
89
- * Bryan Ford for his seminal work on Packrat Parsers.
90
- * The editors of Lambda the Ultimate, where I discovered parsing expression grammars.
@@ -1,51 +0,0 @@
1
- #Pitfalls
2
- ##Left Recursion
3
- An weakness shared by all recursive descent parsers is the inability to parse left-recursive rules. Consider the following rule:
4
-
5
- rule left_recursive
6
- left_recursive 'a' / 'a'
7
- end
8
-
9
- Logically it should match a list of 'a' characters. But it never consumes anything, because attempting to recognize `left_recursive` begins by attempting to recognize `left_recursive`, and so goes an infinite recursion. There's always a way to eliminate these types of structures from your grammar. There's a mechanistic transformation called _left factorization_ that can eliminate it, but it isn't always pretty, especially in combination with automatically constructed syntax trees. So far, I have found more thoughtful ways around the problem. For instance, in the interpreter example I interpret inherently left-recursive function application right recursively in syntax, then correct the directionality in my semantic interpretation. You may have to be clever.
10
-
11
- #Advanced Techniques
12
- Here are a few interesting problems I've encountered. I figure sharing them may give you insight into how these types of issues are addressed with the tools of parsing expressions.
13
-
14
- ##Matching a String
15
-
16
- rule string
17
- '"' (!'"' . / '\"')* '"'
18
- end
19
-
20
- This expression says: Match a quote, then zero or more of any character but a quote or an escaped quote followed by a quote. Lookahead assertions are essential for these types of problems.
21
-
22
- ##Matching Nested Structures With Non-Unique Delimeters
23
- Say I want to parse a diabolical wiki syntax in which the following interpretations apply.
24
-
25
- ** *hello* ** --> <strong><em>hello</em></strong>
26
- * **hello** * --> <em><strong>hello</strong></em>
27
-
28
- rule strong
29
- '**' (em / !'*' . / '\*')+ '**'
30
- end
31
-
32
- rule em
33
- '**' (strong / !'*' . / '\*')+ '**'
34
- end
35
-
36
- Emphasized text is allowed within strong text by virtue of `em` being the first alternative. Since `em` will only successfully parse if a matching `*` is found, it is permitted, but other than that, no `*` characters are allowed unless they are escaped.
37
-
38
- ##Matching a Keyword But Not Words Prefixed Therewith
39
- Say I want to consider a given string a characters only when it occurs in isolation. Lets use the `end` keyword as an example. We don't want the prefix of `'enders_game'` to be considered a keyword. A naiive implementation might be the following.
40
-
41
- rule end_keyword
42
- 'end' &space
43
- end
44
-
45
- This says that `'end'` must be followed by a space, but this space is not consumed as part of the matching of `keyword`. This works in most cases, but is actually incorrect. What if `end` occurs at the end of the buffer? In that case, it occurs in isolation but will not match the above expression. What we really mean is that `'end'` cannot be followed by a _non-space_ character.
46
-
47
- rule end_keyword
48
- 'end' !(!' ' .)
49
- end
50
-
51
- In general, when the syntax gets tough, it helps to focus on what you really mean. A keyword is a character not followed by another character that isn't a space.
@@ -1,189 +0,0 @@
1
- #Semantic Interpretation
2
- Lets use the below grammar as an example. It describes parentheses wrapping a single character to an arbitrary depth.
3
-
4
- grammar ParenLanguage
5
- rule parenthesized_letter
6
- '(' parenthesized_letter ')'
7
- /
8
- [a-z]
9
- end
10
- end
11
-
12
- Matches:
13
-
14
- * `'a'`
15
- * `'(a)'`
16
- * `'((a))'`
17
- * etc.
18
-
19
-
20
- Output from a parser for this grammar looks like this:
21
-
22
- ![Tree Returned By ParenLanguageParser](./images/paren_language_output.png)
23
-
24
- This is a parse tree whose nodes are instances of `Treetop::Runtime::SyntaxNode`. What if we could define methods on these node objects? We would then have an object-oriented program whose structure corresponded to the structure of our language. Treetop provides two techniques for doing just this.
25
-
26
- ##Associating Methods with Node-Instantiating Expressions
27
- Sequences and all types of terminals are node-instantiating expressions. When they match, they create instances of `Treetop::Runtime::SyntaxNode`. Methods can be added to these nodes in the following ways:
28
-
29
- ###Inline Method Definition
30
- Methods can be added to the nodes instantiated by the successful match of an expression
31
-
32
- grammar ParenLanguage
33
- rule parenthesized_letter
34
- '(' parenthesized_letter ')' {
35
- def depth
36
- parenthesized_letter.depth + 1
37
- end
38
- }
39
- /
40
- [a-z] {
41
- def depth
42
- 0
43
- end
44
- }
45
- end
46
- end
47
-
48
- Note that each alternative expression is followed by a block containing a method definition. A `depth` method is defined on both expressions. The recursive `depth` method defined in the block following the first expression determines the depth of the nested parentheses and adds one to it. The base case is implemented in the block following the second expression; a single character has a depth of 0.
49
-
50
-
51
- ###Custom `SyntaxNode` Subclass Declarations
52
- You can instruct the parser to instantiate a custom subclass of Treetop::Runtime::SyntaxNode for an expression by following it by the name of that class enclosed in angle brackets (`<>`). The above inline method definitions could have been moved out into a single class like so.
53
-
54
- # in .treetop file
55
- grammar ParenLanguage
56
- rule parenthesized_letter
57
- '(' parenthesized_letter ')' <ParenNode>
58
- /
59
- [a-z] <ParenNode>
60
- end
61
- end
62
-
63
- # in separate .rb file
64
- class ParenNode < Treetop::Runtime::SyntaxNode
65
- def depth
66
- if nonterminal?
67
- parenthesized_letter.depth + 1
68
- else
69
- 0
70
- end
71
- end
72
- end
73
-
74
- ##Automatic Extension of Results
75
- Nonterminal and ordered choice expressions do not instantiate new nodes, but rather pass through nodes that are instantiated by other expressions. They can extend nodes they propagate with anonymous or declared modules, using similar constructs used with expressions that instantiate their own syntax nodes.
76
-
77
- ###Extending a Propagated Node with an Anonymous Module
78
- rule parenthesized_letter
79
- ('(' parenthesized_letter ')' / [a-z]) {
80
- def depth
81
- if nonterminal?
82
- parenthesized_letter.depth + 1
83
- else
84
- 0
85
- end
86
- end
87
- }
88
- end
89
-
90
- The parenthesized choice above can result in a node matching either of the two choices. Than node will be extended with methods defined in the subsequent block. Note that a choice must always be parenthesized to be associated with a following block.
91
-
92
- ###Extending A Propagated Node with a Declared Module
93
- # in .treetop file
94
- rule parenthesized_letter
95
- ('(' parenthesized_letter ')' / [a-z]) <ParenNode>
96
- end
97
-
98
- # in separate .rb file
99
- module ParenNode
100
- def depth
101
- if nonterminal?
102
- parenthesized_letter.depth + 1
103
- else
104
- 0
105
- end
106
- end
107
- end
108
-
109
- Here the result is extended with the `ParenNode` module. Note the previous example for node-instantiating expressions, the constant in the declaration must be a module because the result is extended with it.
110
-
111
- ##Automatically-Defined Element Accessor Methods
112
- ###Default Accessors
113
- Nodes instantiated upon the matching of sequences have methods automatically defined for any nonterminals in the sequence.
114
-
115
- rule abc
116
- a b c {
117
- def to_s
118
- a.to_s + b.to_s + c.to_s
119
- end
120
- }
121
- end
122
-
123
- In the above code, the `to_s` method calls automatically-defined element accessors for the nodes returned by parsing nonterminals `a`, `b`, and `c`.
124
-
125
- ###Labels
126
- Subexpressions can be given an explicit label to have an element accessor method defined for them. This is useful in cases of ambiguity between two references to the same nonterminal or when you need to access an unnamed subexpression.
127
-
128
- rule labels
129
- first_letter:[a-z] rest_letters:(', ' letter:[a-z])* {
130
- def letters
131
- [first_letter] + rest_letters.map do |comma_and_letter|
132
- comma_and_letter.letter
133
- end
134
- end
135
- }
136
- end
137
-
138
- The above grammar uses label-derived accessors to determine the letters in a comma-delimited list of letters. The labeled expressions _could_ have been extracted to their own rules, but if they aren't used elsewhere, labels still enable them to be referenced by a name within the expression's methods.
139
-
140
- ###Overriding Element Accessors
141
- The module containing automatically defined element accessor methods is an ancestor of the module in which you define your own methods, meaning you can override them with access to the `super` keyword. Here's an example of how this fact can improve the readability of the example above.
142
-
143
- rule labels
144
- first_letter:[a-z] rest_letters:(', ' letter:[a-z])* {
145
- def letters
146
- [first_letter] + rest_letters
147
- end
148
-
149
- def rest_letters
150
- super.map { |comma_and_letter| comma_and_letter.letter }
151
- end
152
- }
153
- end
154
-
155
-
156
- ##Methods Available on `Treetop::Runtime::SyntaxNode`
157
-
158
- <table>
159
- <tr>
160
- <td>
161
- <code>terminal?</code>
162
- </td>
163
- <td>
164
- Was this node produced by the matching of a terminal symbol?
165
- </td>
166
- </tr>
167
- <tr>
168
- <td>
169
- <code>nonterminal?</code>
170
- </td>
171
- <td>
172
- Was this node produced by the matching of a nonterminal symbol?
173
- </td>
174
- <tr>
175
- <td>
176
- <code>text_value</code>
177
- </td>
178
- <td>
179
- The substring of the input represented by this node.
180
- </td>
181
- <tr>
182
- <td>
183
- <code>elements</code>
184
- </td>
185
- <td>
186
- Available only on nonterminal nodes, returns the nodes parsed by the elements of the matched sequence.
187
- </td>
188
- </tr>
189
- </table>