yarp 0.12.0 → 0.13.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (115) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +29 -8
  3. data/CONTRIBUTING.md +2 -2
  4. data/Makefile +5 -5
  5. data/README.md +11 -12
  6. data/config.yml +6 -2
  7. data/docs/build_system.md +21 -21
  8. data/docs/building.md +4 -4
  9. data/docs/configuration.md +25 -21
  10. data/docs/design.md +2 -2
  11. data/docs/encoding.md +17 -17
  12. data/docs/fuzzing.md +4 -4
  13. data/docs/heredocs.md +3 -3
  14. data/docs/mapping.md +94 -94
  15. data/docs/ripper.md +4 -4
  16. data/docs/ruby_api.md +11 -11
  17. data/docs/serialization.md +17 -16
  18. data/docs/testing.md +6 -6
  19. data/ext/prism/api_node.c +4725 -0
  20. data/ext/{yarp → prism}/api_pack.c +82 -82
  21. data/ext/{yarp → prism}/extconf.rb +13 -13
  22. data/ext/{yarp → prism}/extension.c +175 -168
  23. data/ext/prism/extension.h +18 -0
  24. data/include/prism/ast.h +1932 -0
  25. data/include/prism/defines.h +45 -0
  26. data/include/prism/diagnostic.h +231 -0
  27. data/include/{yarp/enc/yp_encoding.h → prism/enc/pm_encoding.h} +40 -40
  28. data/include/prism/node.h +41 -0
  29. data/include/prism/pack.h +141 -0
  30. data/include/{yarp → prism}/parser.h +143 -142
  31. data/include/prism/regexp.h +19 -0
  32. data/include/prism/unescape.h +48 -0
  33. data/include/prism/util/pm_buffer.h +51 -0
  34. data/include/{yarp/util/yp_char.h → prism/util/pm_char.h} +20 -20
  35. data/include/{yarp/util/yp_constant_pool.h → prism/util/pm_constant_pool.h} +26 -22
  36. data/include/{yarp/util/yp_list.h → prism/util/pm_list.h} +21 -21
  37. data/include/prism/util/pm_memchr.h +14 -0
  38. data/include/{yarp/util/yp_newline_list.h → prism/util/pm_newline_list.h} +11 -11
  39. data/include/prism/util/pm_state_stack.h +24 -0
  40. data/include/{yarp/util/yp_string.h → prism/util/pm_string.h} +20 -20
  41. data/include/prism/util/pm_string_list.h +25 -0
  42. data/include/{yarp/util/yp_strpbrk.h → prism/util/pm_strpbrk.h} +7 -7
  43. data/include/prism/version.h +4 -0
  44. data/include/prism.h +82 -0
  45. data/lib/prism/compiler.rb +465 -0
  46. data/lib/prism/debug.rb +157 -0
  47. data/lib/{yarp/desugar_visitor.rb → prism/desugar_compiler.rb} +4 -2
  48. data/lib/prism/dispatcher.rb +2051 -0
  49. data/lib/prism/dsl.rb +750 -0
  50. data/lib/{yarp → prism}/ffi.rb +66 -67
  51. data/lib/{yarp → prism}/lex_compat.rb +40 -43
  52. data/lib/{yarp/mutation_visitor.rb → prism/mutation_compiler.rb} +3 -3
  53. data/lib/{yarp → prism}/node.rb +2012 -2593
  54. data/lib/prism/node_ext.rb +55 -0
  55. data/lib/prism/node_inspector.rb +68 -0
  56. data/lib/{yarp → prism}/pack.rb +1 -1
  57. data/lib/{yarp → prism}/parse_result/comments.rb +1 -1
  58. data/lib/{yarp → prism}/parse_result/newlines.rb +1 -1
  59. data/lib/prism/parse_result.rb +266 -0
  60. data/lib/{yarp → prism}/pattern.rb +14 -14
  61. data/lib/{yarp → prism}/ripper_compat.rb +5 -5
  62. data/lib/{yarp → prism}/serialize.rb +12 -7
  63. data/lib/prism/visitor.rb +470 -0
  64. data/lib/prism.rb +64 -0
  65. data/lib/yarp.rb +2 -614
  66. data/src/diagnostic.c +213 -208
  67. data/src/enc/pm_big5.c +52 -0
  68. data/src/enc/pm_euc_jp.c +58 -0
  69. data/src/enc/{yp_gbk.c → pm_gbk.c} +16 -16
  70. data/src/enc/pm_shift_jis.c +56 -0
  71. data/src/enc/{yp_tables.c → pm_tables.c} +69 -69
  72. data/src/enc/{yp_unicode.c → pm_unicode.c} +40 -40
  73. data/src/enc/pm_windows_31j.c +56 -0
  74. data/src/node.c +1293 -1233
  75. data/src/pack.c +247 -247
  76. data/src/prettyprint.c +1479 -1479
  77. data/src/{yarp.c → prism.c} +5205 -5083
  78. data/src/regexp.c +132 -132
  79. data/src/serialize.c +1121 -1121
  80. data/src/token_type.c +169 -167
  81. data/src/unescape.c +106 -87
  82. data/src/util/pm_buffer.c +103 -0
  83. data/src/util/{yp_char.c → pm_char.c} +72 -72
  84. data/src/util/{yp_constant_pool.c → pm_constant_pool.c} +85 -64
  85. data/src/util/{yp_list.c → pm_list.c} +10 -10
  86. data/src/util/{yp_memchr.c → pm_memchr.c} +6 -4
  87. data/src/util/{yp_newline_list.c → pm_newline_list.c} +21 -21
  88. data/src/util/{yp_state_stack.c → pm_state_stack.c} +4 -4
  89. data/src/util/{yp_string.c → pm_string.c} +38 -38
  90. data/src/util/pm_string_list.c +29 -0
  91. data/src/util/{yp_strncasecmp.c → pm_strncasecmp.c} +1 -1
  92. data/src/util/{yp_strpbrk.c → pm_strpbrk.c} +8 -8
  93. data/yarp.gemspec +68 -59
  94. metadata +70 -61
  95. data/ext/yarp/api_node.c +0 -4728
  96. data/ext/yarp/extension.h +0 -18
  97. data/include/yarp/ast.h +0 -1929
  98. data/include/yarp/defines.h +0 -45
  99. data/include/yarp/diagnostic.h +0 -226
  100. data/include/yarp/node.h +0 -42
  101. data/include/yarp/pack.h +0 -141
  102. data/include/yarp/regexp.h +0 -19
  103. data/include/yarp/unescape.h +0 -44
  104. data/include/yarp/util/yp_buffer.h +0 -51
  105. data/include/yarp/util/yp_memchr.h +0 -14
  106. data/include/yarp/util/yp_state_stack.h +0 -24
  107. data/include/yarp/util/yp_string_list.h +0 -25
  108. data/include/yarp/version.h +0 -4
  109. data/include/yarp.h +0 -82
  110. data/src/enc/yp_big5.c +0 -52
  111. data/src/enc/yp_euc_jp.c +0 -58
  112. data/src/enc/yp_shift_jis.c +0 -56
  113. data/src/enc/yp_windows_31j.c +0 -56
  114. data/src/util/yp_buffer.c +0 -101
  115. data/src/util/yp_string_list.c +0 -29
data/docs/mapping.md CHANGED
@@ -1,117 +1,117 @@
1
1
  # Mapping
2
2
 
3
- When considering the previous CRuby parser versus YARP, this document should be helpful to understand how various concepts are mapped.
3
+ When considering the previous CRuby parser versus prism, this document should be helpful to understand how various concepts are mapped.
4
4
 
5
5
  ## Nodes
6
6
 
7
- The following table shows how the various CRuby nodes are mapped to YARP nodes.
7
+ The following table shows how the various CRuby nodes are mapped to prism nodes.
8
8
 
9
- | CRuby | YARP |
9
+ | CRuby | prism |
10
10
  | --- | --- |
11
11
  | `NODE_SCOPE` | |
12
12
  | `NODE_BLOCK` | |
13
- | `NODE_IF` | `YP_IF_NODE` |
14
- | `NODE_UNLESS` | `YP_UNLESS_NODE` |
15
- | `NODE_CASE` | `YP_CASE_NODE` |
16
- | `NODE_CASE2` | `YP_CASE_NODE` (with a null predicate) |
13
+ | `NODE_IF` | `PM_IF_NODE` |
14
+ | `NODE_UNLESS` | `PM_UNLESS_NODE` |
15
+ | `NODE_CASE` | `PM_CASE_NODE` |
16
+ | `NODE_CASE2` | `PM_CASE_NODE` (with a null predicate) |
17
17
  | `NODE_CASE3` | |
18
- | `NODE_WHEN` | `YP_WHEN_NODE` |
19
- | `NODE_IN` | `YP_IN_NODE` |
20
- | `NODE_WHILE` | `YP_WHILE_NODE` |
21
- | `NODE_UNTIL` | `YP_UNTIL_NODE` |
22
- | `NODE_ITER` | `YP_CALL_NODE` (with a non-null block) |
23
- | `NODE_FOR` | `YP_FOR_NODE` |
24
- | `NODE_FOR_MASGN` | `YP_FOR_NODE` (with a multi-write node as the index) |
25
- | `NODE_BREAK` | `YP_BREAK_NODE` |
26
- | `NODE_NEXT` | `YP_NEXT_NODE` |
27
- | `NODE_REDO` | `YP_REDO_NODE` |
28
- | `NODE_RETRY` | `YP_RETRY_NODE` |
29
- | `NODE_BEGIN` | `YP_BEGIN_NODE` |
30
- | `NODE_RESCUE` | `YP_RESCUE_NODE` |
18
+ | `NODE_WHEN` | `PM_WHEN_NODE` |
19
+ | `NODE_IN` | `PM_IN_NODE` |
20
+ | `NODE_WHILE` | `PM_WHILE_NODE` |
21
+ | `NODE_UNTIL` | `PM_UNTIL_NODE` |
22
+ | `NODE_ITER` | `PM_CALL_NODE` (with a non-null block) |
23
+ | `NODE_FOR` | `PM_FOR_NODE` |
24
+ | `NODE_FOR_MASGN` | `PM_FOR_NODE` (with a multi-write node as the index) |
25
+ | `NODE_BREAK` | `PM_BREAK_NODE` |
26
+ | `NODE_NEXT` | `PM_NEXT_NODE` |
27
+ | `NODE_REDO` | `PM_REDO_NODE` |
28
+ | `NODE_RETRY` | `PM_RETRY_NODE` |
29
+ | `NODE_BEGIN` | `PM_BEGIN_NODE` |
30
+ | `NODE_RESCUE` | `PM_RESCUE_NODE` |
31
31
  | `NODE_RESBODY` | |
32
- | `NODE_ENSURE` | `YP_ENSURE_NODE` |
33
- | `NODE_AND` | `YP_AND_NODE` |
34
- | `NODE_OR` | `YP_OR_NODE` |
35
- | `NODE_MASGN` | `YP_MULTI_WRITE_NODE` |
36
- | `NODE_LASGN` | `YP_LOCAL_VARIABLE_WRITE_NODE` |
37
- | `NODE_DASGN` | `YP_LOCAL_VARIABLE_WRITE_NODE` |
38
- | `NODE_GASGN` | `YP_GLOBAL_VARIABLE_WRITE_NODE` |
39
- | `NODE_IASGN` | `YP_INSTANCE_VARIABLE_WRITE_NODE` |
40
- | `NODE_CDECL` | `YP_CONSTANT_PATH_WRITE_NODE` |
41
- | `NODE_CVASGN` | `YP_CLASS_VARIABLE_WRITE_NODE` |
32
+ | `NODE_ENSURE` | `PM_ENSURE_NODE` |
33
+ | `NODE_AND` | `PM_AND_NODE` |
34
+ | `NODE_OR` | `PM_OR_NODE` |
35
+ | `NODE_MASGN` | `PM_MULTI_WRITE_NODE` |
36
+ | `NODE_LASGN` | `PM_LOCAL_VARIABLE_WRITE_NODE` |
37
+ | `NODE_DASGN` | `PM_LOCAL_VARIABLE_WRITE_NODE` |
38
+ | `NODE_GASGN` | `PM_GLOBAL_VARIABLE_WRITE_NODE` |
39
+ | `NODE_IASGN` | `PM_INSTANCE_VARIABLE_WRITE_NODE` |
40
+ | `NODE_CDECL` | `PM_CONSTANT_PATH_WRITE_NODE` |
41
+ | `NODE_CVASGN` | `PM_CLASS_VARIABLE_WRITE_NODE` |
42
42
  | `NODE_OP_ASGN1` | |
43
43
  | `NODE_OP_ASGN2` | |
44
- | `NODE_OP_ASGN_AND` | `YP_OPERATOR_AND_ASSIGNMENT_NODE` |
45
- | `NODE_OP_ASGN_OR` | `YP_OPERATOR_OR_ASSIGNMENT_NODE` |
44
+ | `NODE_OP_ASGN_AND` | `PM_OPERATOR_AND_ASSIGNMENT_NODE` |
45
+ | `NODE_OP_ASGN_OR` | `PM_OPERATOR_OR_ASSIGNMENT_NODE` |
46
46
  | `NODE_OP_CDECL` | |
47
- | `NODE_CALL` | `YP_CALL_NODE` |
48
- | `NODE_OPCALL` | `YP_CALL_NODE` (with an operator as the method) |
49
- | `NODE_FCALL` | `YP_CALL_NODE` (with a null receiver and parentheses) |
50
- | `NODE_VCALL` | `YP_CALL_NODE` (with a null receiver and parentheses or arguments) |
51
- | `NODE_QCALL` | `YP_CALL_NODE` (with a &. operator) |
52
- | `NODE_SUPER` | `YP_SUPER_NODE` |
53
- | `NODE_ZSUPER` | `YP_FORWARDING_SUPER_NODE` |
54
- | `NODE_LIST` | `YP_ARRAY_NODE` |
55
- | `NODE_ZLIST` | `YP_ARRAY_NODE` (with no child elements) |
56
- | `NODE_VALUES` | `YP_ARGUMENTS_NODE` |
57
- | `NODE_HASH` | `YP_HASH_NODE` |
58
- | `NODE_RETURN` | `YP_RETURN_NODE` |
59
- | `NODE_YIELD` | `YP_YIELD_NODE` |
60
- | `NODE_LVAR` | `YP_LOCAL_VARIABLE_READ_NODE` |
61
- | `NODE_DVAR` | `YP_LOCAL_VARIABLE_READ_NODE` |
62
- | `NODE_GVAR` | `YP_GLOBAL_VARIABLE_READ_NODE` |
63
- | `NODE_IVAR` | `YP_INSTANCE_VARIABLE_READ_NODE` |
64
- | `NODE_CONST` | `YP_CONSTANT_PATH_READ_NODE` |
65
- | `NODE_CVAR` | `YP_CLASS_VARIABLE_READ_NODE` |
66
- | `NODE_NTH_REF` | `YP_NUMBERED_REFERENCE_READ_NODE` |
67
- | `NODE_BACK_REF` | `YP_BACK_REFERENCE_READ_NODE` |
47
+ | `NODE_CALL` | `PM_CALL_NODE` |
48
+ | `NODE_OPCALL` | `PM_CALL_NODE` (with an operator as the method) |
49
+ | `NODE_FCALL` | `PM_CALL_NODE` (with a null receiver and parentheses) |
50
+ | `NODE_VCALL` | `PM_CALL_NODE` (with a null receiver and parentheses or arguments) |
51
+ | `NODE_QCALL` | `PM_CALL_NODE` (with a &. operator) |
52
+ | `NODE_SUPER` | `PM_SUPER_NODE` |
53
+ | `NODE_ZSUPER` | `PM_FORWARDING_SUPER_NODE` |
54
+ | `NODE_LIST` | `PM_ARRAY_NODE` |
55
+ | `NODE_ZLIST` | `PM_ARRAY_NODE` (with no child elements) |
56
+ | `NODE_VALUES` | `PM_ARGUMENTS_NODE` |
57
+ | `NODE_HASH` | `PM_HASH_NODE` |
58
+ | `NODE_RETURN` | `PM_RETURN_NODE` |
59
+ | `NODE_YIELD` | `PM_YIELD_NODE` |
60
+ | `NODE_LVAR` | `PM_LOCAL_VARIABLE_READ_NODE` |
61
+ | `NODE_DVAR` | `PM_LOCAL_VARIABLE_READ_NODE` |
62
+ | `NODE_GVAR` | `PM_GLOBAL_VARIABLE_READ_NODE` |
63
+ | `NODE_IVAR` | `PM_INSTANCE_VARIABLE_READ_NODE` |
64
+ | `NODE_CONST` | `PM_CONSTANT_PATH_READ_NODE` |
65
+ | `NODE_CVAR` | `PM_CLASS_VARIABLE_READ_NODE` |
66
+ | `NODE_NTH_REF` | `PM_NUMBERED_REFERENCE_READ_NODE` |
67
+ | `NODE_BACK_REF` | `PM_BACK_REFERENCE_READ_NODE` |
68
68
  | `NODE_MATCH` | |
69
- | `NODE_MATCH2` | `YP_CALL_NODE` (with regular expression as receiver) |
70
- | `NODE_MATCH3` | `YP_CALL_NODE` (with regular expression as only argument) |
69
+ | `NODE_MATCH2` | `PM_CALL_NODE` (with regular expression as receiver) |
70
+ | `NODE_MATCH3` | `PM_CALL_NODE` (with regular expression as only argument) |
71
71
  | `NODE_LIT` | |
72
- | `NODE_STR` | `YP_STRING_NODE` |
73
- | `NODE_DSTR` | `YP_INTERPOLATED_STRING_NODE` |
74
- | `NODE_XSTR` | `YP_X_STRING_NODE` |
75
- | `NODE_DXSTR` | `YP_INTERPOLATED_X_STRING_NODE` |
76
- | `NODE_EVSTR` | `YP_STRING_INTERPOLATED_NODE` |
77
- | `NODE_DREGX` | `YP_INTERPOLATED_REGULAR_EXPRESSION_NODE` |
72
+ | `NODE_STR` | `PM_STRING_NODE` |
73
+ | `NODE_DSTR` | `PM_INTERPOLATED_STRING_NODE` |
74
+ | `NODE_XSTR` | `PM_X_STRING_NODE` |
75
+ | `NODE_DXSTR` | `PM_INTERPOLATED_X_STRING_NODE` |
76
+ | `NODE_EVSTR` | `PM_STRING_INTERPOLATED_NODE` |
77
+ | `NODE_DREGX` | `PM_INTERPOLATED_REGULAR_EXPRESSION_NODE` |
78
78
  | `NODE_ONCE` | |
79
- | `NODE_ARGS` | `YP_PARAMETERS_NODE` |
79
+ | `NODE_ARGS` | `PM_PARAMETERS_NODE` |
80
80
  | `NODE_ARGS_AUX` | |
81
- | `NODE_OPT_ARG` | `YP_OPTIONAL_PARAMETER_NODE` |
82
- | `NODE_KW_ARG` | `YP_KEYWORD_PARAMETER_NODE` |
83
- | `NODE_POSTARG` | `YP_REQUIRED_PARAMETER_NODE` |
81
+ | `NODE_OPT_ARG` | `PM_OPTIONAL_PARAMETER_NODE` |
82
+ | `NODE_KW_ARG` | `PM_KEYWORD_PARAMETER_NODE` |
83
+ | `NODE_POSTARG` | `PM_REQUIRED_PARAMETER_NODE` |
84
84
  | `NODE_ARGSCAT` | |
85
85
  | `NODE_ARGSPUSH` | |
86
- | `NODE_SPLAT` | `YP_SPLAT_NODE` |
87
- | `NODE_BLOCK_PASS` | `YP_BLOCK_ARGUMENT_NODE` |
88
- | `NODE_DEFN` | `YP_DEF_NODE` (with a null receiver) |
89
- | `NODE_DEFS` | `YP_DEF_NODE` (with a non-null receiver) |
90
- | `NODE_ALIAS` | `YP_ALIAS_NODE` |
91
- | `NODE_VALIAS` | `YP_ALIAS_NODE` (with a global variable first argument) |
92
- | `NODE_UNDEF` | `YP_UNDEF_NODE` |
93
- | `NODE_CLASS` | `YP_CLASS_NODE` |
94
- | `NODE_MODULE` | `YP_MODULE_NODE` |
95
- | `NODE_SCLASS` | `YP_S_CLASS_NODE` |
96
- | `NODE_COLON2` | `YP_CONSTANT_PATH_NODE` |
97
- | `NODE_COLON3` | `YP_CONSTANT_PATH_NODE` (with a null receiver) |
98
- | `NODE_DOT2` | `YP_RANGE_NODE` (with a .. operator) |
99
- | `NODE_DOT3` | `YP_RANGE_NODE` (with a ... operator) |
100
- | `NODE_FLIP2` | `YP_RANGE_NODE` (with a .. operator) |
101
- | `NODE_FLIP3` | `YP_RANGE_NODE` (with a ... operator) |
102
- | `NODE_SELF` | `YP_SELF_NODE` |
103
- | `NODE_NIL` | `YP_NIL_NODE` |
104
- | `NODE_TRUE` | `YP_TRUE_NODE` |
105
- | `NODE_FALSE` | `YP_FALSE_NODE` |
86
+ | `NODE_SPLAT` | `PM_SPLAT_NODE` |
87
+ | `NODE_BLOCK_PASS` | `PM_BLOCK_ARGUMENT_NODE` |
88
+ | `NODE_DEFN` | `PM_DEF_NODE` (with a null receiver) |
89
+ | `NODE_DEFS` | `PM_DEF_NODE` (with a non-null receiver) |
90
+ | `NODE_ALIAS` | `PM_ALIAS_NODE` |
91
+ | `NODE_VALIAS` | `PM_ALIAS_NODE` (with a global variable first argument) |
92
+ | `NODE_UNDEF` | `PM_UNDEF_NODE` |
93
+ | `NODE_CLASS` | `PM_CLASS_NODE` |
94
+ | `NODE_MODULE` | `PM_MODULE_NODE` |
95
+ | `NODE_SCLASS` | `PM_S_CLASS_NODE` |
96
+ | `NODE_COLON2` | `PM_CONSTANT_PATH_NODE` |
97
+ | `NODE_COLON3` | `PM_CONSTANT_PATH_NODE` (with a null receiver) |
98
+ | `NODE_DOT2` | `PM_RANGE_NODE` (with a .. operator) |
99
+ | `NODE_DOT3` | `PM_RANGE_NODE` (with a ... operator) |
100
+ | `NODE_FLIP2` | `PM_RANGE_NODE` (with a .. operator) |
101
+ | `NODE_FLIP3` | `PM_RANGE_NODE` (with a ... operator) |
102
+ | `NODE_SELF` | `PM_SELF_NODE` |
103
+ | `NODE_NIL` | `PM_NIL_NODE` |
104
+ | `NODE_TRUE` | `PM_TRUE_NODE` |
105
+ | `NODE_FALSE` | `PM_FALSE_NODE` |
106
106
  | `NODE_ERRINFO` | |
107
- | `NODE_DEFINED` | `YP_DEFINED_NODE` |
108
- | `NODE_POSTEXE` | `YP_POST_EXECUTION_NODE` |
109
- | `NODE_DSYM` | `YP_INTERPOLATED_SYMBOL_NODE` |
110
- | `NODE_ATTRASGN` | `YP_CALL_NODE` (with a message that ends with =) |
111
- | `NODE_LAMBDA` | `YP_LAMBDA_NODE` |
112
- | `NODE_ARYPTN` | `YP_ARRAY_PATTERN_NODE` |
113
- | `NODE_HSHPTN` | `YP_HASH_PATTERN_NODE` |
114
- | `NODE_FNDPTN` | `YP_FIND_PATTERN_NODE` |
115
- | `NODE_ERROR` | `YP_MISSING_NODE` |
107
+ | `NODE_DEFINED` | `PM_DEFINED_NODE` |
108
+ | `NODE_POSTEXE` | `PM_POST_EXECUTION_NODE` |
109
+ | `NODE_DSYM` | `PM_INTERPOLATED_SYMBOL_NODE` |
110
+ | `NODE_ATTRASGN` | `PM_CALL_NODE` (with a message that ends with =) |
111
+ | `NODE_LAMBDA` | `PM_LAMBDA_NODE` |
112
+ | `NODE_ARYPTN` | `PM_ARRAY_PATTERN_NODE` |
113
+ | `NODE_HSHPTN` | `PM_HASH_PATTERN_NODE` |
114
+ | `NODE_FNDPTN` | `PM_FIND_PATTERN_NODE` |
115
+ | `NODE_ERROR` | `PM_MISSING_NODE` |
116
116
  | `NODE_LAST` | |
117
117
  ```
data/docs/ripper.md CHANGED
@@ -2,12 +2,12 @@
2
2
 
3
3
  To test the parser, we compare against the output from `Ripper`, both for testing the lexer and testing the parser. The lexer test suite is much more feature complete at the moment.
4
4
 
5
- To lex source code using `YARP`, you typically would run `YARP.lex(source)`. If you want to instead get output that `Ripper` would normally produce, you can run `YARP.lex_compat(source)`. This will produce tokens that should be equivalent to `Ripper`.
5
+ To lex source code using `prism`, you typically would run `Prism.lex(source)`. If you want to instead get output that `Ripper` would normally produce, you can run `Prism.lex_compat(source)`. This will produce tokens that should be equivalent to `Ripper`.
6
6
 
7
- To parse source code using `YARP`, you typically would run `YARP.parse(source)`. If you want to instead using the `Ripper` streaming interface, you can inherit from `YARP::RipperCompat` and override the `on_*` methods. This will produce a syntax tree that should be equivalent to `Ripper`. That would look like:
7
+ To parse source code using `prism`, you typically would run `Prism.parse(source)`. If you want to instead using the `Ripper` streaming interface, you can inherit from `Prism::RipperCompat` and override the `on_*` methods. This will produce a syntax tree that should be equivalent to `Ripper`. That would look like:
8
8
 
9
9
  ```ruby
10
- class ArithmeticRipper < YARP::RipperCompat
10
+ class ArithmeticRipper < Prism::RipperCompat
11
11
  def on_binary(left, operator, right)
12
12
  left.public_send(operator, right)
13
13
  end
@@ -33,4 +33,4 @@ end
33
33
  ArithmeticRipper.new("1 + 2 - 3").parse # => [0]
34
34
  ```
35
35
 
36
- There are also APIs for building trees similar to the s-expression builders in `Ripper`. The method names are the same. These include `YARP::RipperCompat.sexp_raw(source)` and `YARP::RipperCompat.sexp(source)`.
36
+ There are also APIs for building trees similar to the s-expression builders in `Ripper`. The method names are the same. These include `Prism::RipperCompat.sexp_raw(source)` and `Prism::RipperCompat.sexp(source)`.
data/docs/ruby_api.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Ruby API
2
2
 
3
- The `yarp` gem provides a Ruby API for accessing the syntax tree.
3
+ The `prism` gem provides a Ruby API for accessing the syntax tree.
4
4
 
5
5
  For the most part, the API for accessing the tree mirrors that found in the [Syntax Tree](https://github.com/ruby-syntax-tree/syntax_tree) project. This means:
6
6
 
@@ -9,17 +9,17 @@ For the most part, the API for accessing the tree mirrors that found in the [Syn
9
9
  * Nodes respond to the pattern matching interfaces `#deconstruct` and `#deconstruct_keys`
10
10
 
11
11
  Every entry in `config.yml` will generate a Ruby class as well as the code that builds the nodes themselves.
12
- Creating a syntax tree involves calling one of the class methods on the `YARP` module.
12
+ Creating a syntax tree involves calling one of the class methods on the `Prism` module.
13
13
  The full API is documented below.
14
14
 
15
15
  ## API
16
16
 
17
- * `YARP.dump(source, filepath)` - parse the syntax tree corresponding to the given source string and filepath, and serialize it to a string. Filepath can be nil.
18
- * `YARP.dump_file(filepath)` - parse the syntax tree corresponding to the given source file and serialize it to a string
19
- * `YARP.lex(source)` - parse the tokens corresponding to the given source string and return them as an array within a parse result
20
- * `YARP.lex_file(filepath)` - parse the tokens corresponding to the given source file and return them as an array within a parse result
21
- * `YARP.parse(source)` - parse the syntax tree corresponding to the given source string and return it within a parse result
22
- * `YARP.parse_file(filepath)` - parse the syntax tree corresponding to the given source file and return it within a parse result
23
- * `YARP.parse_lex(source)` - parse the syntax tree corresponding to the given source string and return it within a parse result, along with the tokens
24
- * `YARP.parse_lex_file(filepath)` - parse the syntax tree corresponding to the given source file and return it within a parse result, along with the tokens
25
- * `YARP.load(source, serialized)` - load the serialized syntax tree using the source as a reference into a syntax tree
17
+ * `Prism.dump(source, filepath)` - parse the syntax tree corresponding to the given source string and filepath, and serialize it to a string. Filepath can be nil.
18
+ * `Prism.dump_file(filepath)` - parse the syntax tree corresponding to the given source file and serialize it to a string
19
+ * `Prism.lex(source)` - parse the tokens corresponding to the given source string and return them as an array within a parse result
20
+ * `Prism.lex_file(filepath)` - parse the tokens corresponding to the given source file and return them as an array within a parse result
21
+ * `Prism.parse(source)` - parse the syntax tree corresponding to the given source string and return it within a parse result
22
+ * `Prism.parse_file(filepath)` - parse the syntax tree corresponding to the given source file and return it within a parse result
23
+ * `Prism.parse_lex(source)` - parse the syntax tree corresponding to the given source string and return it within a parse result, along with the tokens
24
+ * `Prism.parse_lex_file(filepath)` - parse the syntax tree corresponding to the given source file and return it within a parse result, along with the tokens
25
+ * `Prism.load(source, serialized)` - load the serialized syntax tree using the source as a reference into a syntax tree
@@ -1,6 +1,6 @@
1
1
  # Serialization
2
2
 
3
- YARP ships with the ability to serialize a syntax tree to a single string.
3
+ Prism ships with the ability to serialize a syntax tree to a single string.
4
4
  The string can then be deserialized back into a syntax tree using a language other than C.
5
5
  This is useful for using the parsing logic in other tools without having to write a parser in that language.
6
6
  The syntax tree still requires a copy of the original source, as for the most part it just contains byte offsets into the source string.
@@ -50,7 +50,7 @@ The comment type is one of:
50
50
  ## Structure
51
51
 
52
52
  The serialized string representing the syntax tree is composed of three parts: the header, the body, and the constant pool.
53
- The header contains information like the version of YARP that serialized the tree.
53
+ The header contains information like the version of prism that serialized the tree.
54
54
  The body contains the actual nodes in the tree.
55
55
  The constant pool contains constants that were interned while parsing.
56
56
 
@@ -58,10 +58,11 @@ The header is structured like the following table:
58
58
 
59
59
  | # bytes | field |
60
60
  | --- | --- |
61
- | `4` | "YARP" |
61
+ | `5` | "PRISM" |
62
62
  | `1` | major version number |
63
63
  | `1` | minor version number |
64
64
  | `1` | patch version number |
65
+ | `1` | 1 indicates only semantics fields were serialized, 0 indicates all fields were serialized (including location fields) |
65
66
  | string | the encoding name |
66
67
  | varint | number of comments |
67
68
  | comment* | comments |
@@ -116,42 +117,42 @@ After the constant pool, the contents of the owned constants are serialized. Thi
116
117
  The relevant APIs and struct definitions are listed below:
117
118
 
118
119
  ```c
119
- // A yp_buffer_t is a simple memory buffer that stores data in a contiguous
120
+ // A pm_buffer_t is a simple memory buffer that stores data in a contiguous
120
121
  // block of memory. It is used to store the serialized representation of a
121
- // YARP tree.
122
+ // prism tree.
122
123
  typedef struct {
123
124
  char *value;
124
125
  size_t length;
125
126
  size_t capacity;
126
- } yp_buffer_t;
127
+ } pm_buffer_t;
127
128
 
128
- // Initialize a yp_buffer_t with its default values.
129
- bool yp_buffer_init(yp_buffer_t *);
129
+ // Initialize a pm_buffer_t with its default values.
130
+ bool pm_buffer_init(pm_buffer_t *);
130
131
 
131
132
  // Free the memory associated with the buffer.
132
- void yp_buffer_free(yp_buffer_t *);
133
+ void pm_buffer_free(pm_buffer_t *);
133
134
 
134
135
  // Parse and serialize the AST represented by the given source to the given
135
136
  // buffer.
136
- void yp_parse_serialize(const uint8_t *source, size_t length, yp_buffer_t *buffer, const char *metadata);
137
+ void pm_parse_serialize(const uint8_t *source, size_t length, pm_buffer_t *buffer, const char *metadata);
137
138
  ```
138
139
 
139
- Typically you would use a stack-allocated `yp_buffer_t` and call `yp_parse_serialize`, as in:
140
+ Typically you would use a stack-allocated `pm_buffer_t` and call `pm_parse_serialize`, as in:
140
141
 
141
142
  ```c
142
143
  void
143
144
  serialize(const uint8_t *source, size_t length) {
144
- yp_buffer_t buffer;
145
- if (!yp_buffer_init(&buffer)) return;
145
+ pm_buffer_t buffer;
146
+ if (!pm_buffer_init(&buffer)) return;
146
147
 
147
- yp_parse_serialize(source, length, &buffer, NULL);
148
+ pm_parse_serialize(source, length, &buffer, NULL);
148
149
  // Do something with the serialized string.
149
150
 
150
- yp_buffer_free(&buffer);
151
+ pm_buffer_free(&buffer);
151
152
  }
152
153
  ```
153
154
 
154
- The final argument to `yp_parse_serialize` controls the metadata of the source.
155
+ The final argument to `pm_parse_serialize` controls the metadata of the source.
155
156
  This includes the filepath that the source is associated with, and any nested local variables scopes that are necessary to properly parse the file (in the case of parsing an `eval`).
156
157
  Note that no `varint` are used here to make it easier to produce the metadata for the caller, and also serialized size is less important here.
157
158
  The metadata is a serialized format itself, and is structured as follows:
data/docs/testing.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Testing
2
2
 
3
- This document explains how to test YARP, both locally, and against existing test suites.
3
+ This document explains how to test prism, both locally, and against existing test suites.
4
4
 
5
5
  ## Test suite
6
6
 
@@ -8,13 +8,13 @@ This document explains how to test YARP, both locally, and against existing test
8
8
 
9
9
  ### Unit tests
10
10
 
11
- These test specific YARP implementation details like comments, errors, and regular expressions. There are corresponding files for each thing being tested (like `test/errors_test.rb`).
11
+ These test specific prism implementation details like comments, errors, and regular expressions. There are corresponding files for each thing being tested (like `test/errors_test.rb`).
12
12
 
13
13
  ### Snapshot tests
14
14
 
15
- Snapshot tests ensure that parsed output is equivalent to previous parsed output. There are many categorized examples of valid syntax within the `test/yarp/fixtures/` directory. When the test suite runs, it will parse all of this syntax, and compare it against corresponding files in the `test/yarp/snapshots/` directory. For example, `test/yarp/fixtures/strings.txt` has a corresponding `test/yarp/snapshots/strings.txt`.
15
+ Snapshot tests ensure that parsed output is equivalent to previous parsed output. There are many categorized examples of valid syntax within the `test/prism/fixtures/` directory. When the test suite runs, it will parse all of this syntax, and compare it against corresponding files in the `test/prism/snapshots/` directory. For example, `test/prism/fixtures/strings.txt` has a corresponding `test/prism/snapshots/strings.txt`.
16
16
 
17
- If the parsed files do not match, it will raise an error. If there is not a corresponding file in the `test/yarp/snapshots/` directory, one will be created so that it exists for the next test run.
17
+ If the parsed files do not match, it will raise an error. If there is not a corresponding file in the `test/prism/snapshots/` directory, one will be created so that it exists for the next test run.
18
18
 
19
19
  ### Testing against repositories
20
20
 
@@ -24,7 +24,7 @@ To test the parser against a repository, you can run `FILEPATHS='/path/to/reposi
24
24
 
25
25
  As you are working, you will likely want to test your code locally. `test.rb` is ignored by git, so it can be used for local testing. There are also two executables which may help you:
26
26
 
27
- 1. **bin/lex** takes a filepath and compares YARP's lexed output to Ripper's lexed output. It prints any lexed output that doesn't match. It does some minor transformations to the lexed output in order to compare them, like split YARP's heredoc tokens to mirror Ripper's.
27
+ 1. **bin/lex** takes a filepath and compares prism's lexed output to Ripper's lexed output. It prints any lexed output that doesn't match. It does some minor transformations to the lexed output in order to compare them, like split prism's heredoc tokens to mirror Ripper's.
28
28
 
29
29
  ```
30
30
  $ bin/lex test.rb
@@ -42,7 +42,7 @@ $ VERBOSE=1 bin/lex test.rb
42
42
  $ bin/lex -e "1 + 2"
43
43
  ```
44
44
 
45
- 2. **bin/parse** takes a filepath and outputs YARP's parsed node structure generated from reading the file.
45
+ 2. **bin/parse** takes a filepath and outputs prism's parsed node structure generated from reading the file.
46
46
 
47
47
  ```
48
48
  $ bin/parse test.rb