prism 0.18.0 → 0.19.0

data/docs/encoding.md

@@ -16,12 +16,24 @@ The key of the comment can be either "encoding" or "coding". The value of the co
 * `Big5`
 * `Big5-HKSCS`
 * `Big5-UAO`
+* `CESU-8`
 * `CP51932`
 * `CP850`
 * `CP852`
 * `CP855`
+* `CP949`
+* `CP950`
+* `CP951`
+* `Emacs-Mule`
 * `EUC-JP`
+* `eucJP-ms`
+* `EUC-JIS-2004`
+* `EUC-KR`
+* `EUC-TW`
+* `GB12345`
+* `GB18030`
 * `GB1988`
+* `GB2312`
 * `GBK`
 * `IBM437`
 * `IBM720`
@@ -33,6 +45,7 @@ The key of the comment can be either "encoding" or "coding". The value of the co
 * `IBM860`
 * `IBM861`
 * `IBM862`
+* `IBM863`
 * `IBM864`
 * `IBM865`
 * `IBM866`
@@ -53,21 +66,31 @@ The key of the comment can be either "encoding" or "coding". The value of the co
 * `ISO-8859-15`
 * `ISO-8859-16`
 * `KOI8-R`
+* `KOI8-U`
 * `macCentEuro`
 * `macCroatian`
 * `macCyrillic`
 * `macGreek`
 * `macIceland`
+* `MacJapanese`
 * `macRoman`
 * `macRomania`
 * `macThai`
 * `macTurkish`
 * `macUkraine`
 * `Shift_JIS`
+* `SJIS-DoCoMo`
+* `SJIS-KDDI`
+* `SJIS-SoftBank`
+* `stateless-ISO-2022-JP`
+* `stateless-ISO-2022-JP-KDDI`
 * `TIS-620`
 * `US-ASCII`
 * `UTF-8`
 * `UTF8-MAC`
+* `UTF8-DoCoMo`
+* `UTF8-KDDI`
+* `UTF8-SoftBank`
 * `Windows-1250`
 * `Windows-1251`
 * `Windows-1252`
@@ -80,62 +103,7 @@ The key of the comment can be either "encoding" or "coding". The value of the co
 * `Windows-31J`
 * `Windows-874`
 
-For each of these encodings, prism provides a function for checking if the subsequent bytes form an alphabetic or alphanumeric character.
-
-## Support for other encodings
-
-If an encoding is encountered that is not supported by prism, prism will call a user-provided callback function with the name of the encoding if one is provided. That function can be registered with `pm_parser_register_encoding_decode_callback`. The user-provided callback function can then provide a pointer to an encoding struct that contains the requisite functions that prism will use to parse identifiers going forward.
-
-If the user-provided callback function returns `NULL` (the value also provided by the default implementation in case a callback was not registered), an error will be added to the parser's error list and parsing will continue on using the default UTF-8 encoding.
-
-```c
-// This struct defines the functions necessary to implement the encoding
-// interface so we can determine how many bytes the subsequent character takes.
-// Each callback should return the number of bytes, or 0 if the next bytes are
-// invalid for the encoding and type.
-typedef struct {
-    // Return the number of bytes that the next character takes if it is valid
-    // in the encoding. Does not read more than n bytes. It is assumed that n is
-    // at least 1.
-    size_t (*char_width)(const uint8_t *b, ptrdiff_t n);
-
-    // Return the number of bytes that the next character takes if it is valid
-    // in the encoding and is alphabetical. Does not read more than n bytes. It
-    // is assumed that n is at least 1.
-    size_t (*alpha_char)(const uint8_t *b, ptrdiff_t n);
-
-    // Return the number of bytes that the next character takes if it is valid
-    // in the encoding and is alphanumeric. Does not read more than n bytes. It
-    // is assumed that n is at least 1.
-    size_t (*alnum_char)(const uint8_t *b, ptrdiff_t n);
-
-    // Return true if the next character is valid in the encoding and is an
-    // uppercase character. Does not read more than n bytes. It is assumed that
-    // n is at least 1.
-    bool (*isupper_char)(const uint8_t *b, ptrdiff_t n);
-
-    // The name of the encoding. This should correspond to a value that can be
-    // passed to Encoding.find in Ruby.
-    const char *name;
-
-    // Return true if the encoding is a multibyte encoding.
-    bool multibyte;
-} pm_encoding_t;
-
-// When an encoding is encountered that isn't understood by prism, we provide
-// the ability here to call out to a user-defined function to get an encoding
-// struct. If the function returns something that isn't NULL, we set that to
-// our encoding and use it to parse identifiers.
-typedef pm_encoding_t *(*pm_encoding_decode_callback_t)(pm_parser_t *parser, const uint8_t *name, size_t width);
-
-// Register a callback that will be called when prism encounters a magic comment
-// with an encoding referenced that it doesn't understand. The callback should
-// return NULL if it also doesn't understand the encoding or it should return a
-// pointer to a pm_encoding_t struct that contains the functions necessary to
-// parse identifiers.
-PRISM_EXPORTED_FUNCTION void
-pm_parser_register_encoding_decode_callback(pm_parser_t *parser, pm_encoding_decode_callback_t callback);
-```
+For each of these encodings, prism provides functions for checking if the subsequent bytes can be interpreted as a character, and then if that character is alphabetic, alphanumeric, or uppercase.
 
 ## Getting notified when the encoding changes
 

data/docs/local_variable_depth.md

@@ -0,0 +1,229 @@
+# Local variable depth
+
+One feature of Prism is that it resolves local variables as it parses. It's necessary to do this because of ambiguities in the grammar. For example, consider the following code:
+
+```ruby
+foo / bar#/
+```
+
+If `foo` is a local variable, this is a call to `/` with `bar` as an argument, followed by a comment. If it's not a local variable, this is a method call to `foo` with a regular expression argument.
+
+"Depth" refers to the number of visible scopes that Prism has to go up to find the declaration of a local variable.
+Note that this follows the same scoping rules as Ruby, so a local variable is only visible in the scope it is declared in and in blocks nested in that scope.
+The rules for calculating the depth are very important to understand because they may differ from individual Ruby implementations since they are not specified by the language.
+
+Prism uses the minimum number of scopes, i.e., it only creates scopes when necessary semantically, in other words when there must be distinct scopes (which can be observed through `binding.local_variables`).
+That are no "transparent/invisible" scopes in Prism.
+Some Ruby implementations use those for some language constructs and need to adjust by maintaining a depth offset.
+
+Below are the places where a local variable can be written/targeted, along with how the depth is calculated at that point.
+
+## General
+
+In the course of general Ruby code when reading a local variable, the depth is equal to the number of scopes to go up to find the declaration of that variable. For example:
+
+```ruby
+foo = 1
+bar = 2
+baz = 3
+
+foo # depth 0
+tap { bar } # depth 1
+tap { tap { baz } } # depth 2
+```
+
+This also includes writing to a local variable, which could be writing to a local variable that is already declared. For example:
+
+```ruby
+foo = 1
+bar = 2
+
+foo = 3 # depth 0
+tap { bar = 4 } # depth 1
+```
+
+This includes multiple assignment, where the same principle applies. For example:
+
+```ruby
+foo = 1
+bar = 2
+
+foo, bar = 3, 4 # depth 0
+tap { foo, bar = 5, 6 } # depth 1
+```
+
+## `for` loops
+
+`for` loops in Ruby break down to calls to `.each` with a block.
+However in that case local variable reads and writes within the block will be in the same scope as the scope surrounding the `for` and not in a deeper/separate scope (surprising, but this is Ruby semantics).
+For example:
+
+```ruby
+foo = 1
+
+for e in baz
+  foo # depth 0
+  bar = 2 # depth 0
+end
+
+p bar # depth 0, prints 2
+```
+
+The local variable(s) used for the index of the `for` are also at the same depth (as variables inside and outside the `for`):
+
+```ruby
+for e in [1, 2] # depth 0
+  e # depth 0
+end
+
+p e # depth 0, prints 2
+```
+
+## Pattern matching captures
+
+You can target a local variable in a pattern matching expression using capture syntax. Using this syntax, you can target local variables in the current scope or in visible parent scopes. For example:
+
+```ruby
+42 => bar # depth 0
+```
+
+The example above writes to a local variable in the current scope. If the variable is already declared in a higher visible scope, it will be written to that scope instead. For example:
+
+```ruby
+foo = 1
+tap { 42 => foo } # depth 1
+```
+
+## Named capture groups
+
+You can target local variables through named capture groups in regular expressions if they are used on the left-hand side of a `=~` operator. For example:
+
+```ruby
+/(?<foo>\d+)/ =~ "42" # depth 0
+```
+
+This will write to a `foo` local variable. If the variable is already declared in a higher visible scope, it will be written to that scope instead. For example:
+
+```ruby
+foo = 1
+tap { /(?<foo>\d+)/ =~ "42" } # depth 1
+```
+
+## "interpolated once" regular expressions
+
+Regular expressions that interpolate local variables (unrelated to capture group local variables) and have the `o` flag will only interpolate the local variables once for the runtime of the program.
+In CRuby, this is implemented by compiling the regular expression within a nested instruction sequence, which means CRuby thinks the depth is one more than prism does. For example:
+
+```
+$ ruby --dump=insns -e 'foo = 1; /#{foo}/o'
+== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,18)> (catch: false)
+local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
+[ 1] foo@0
+0000 putobject_INT2FIX_1_                                             (   1)[Li]
+0001 setlocal_WC_0                          foo@0
+0003 once                                   block in <main>, <is:0>
+0006 leave
+
+== disasm: #<ISeq:block in <main>@-e:1 (1,9)-(1,18)> (catch: false)
+0000 putobject                              ""                        (   1)
+0002 getlocal_WC_1                          foo@0
+0004 dup
+0005 objtostring                            <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE>
+0007 anytostring
+0008 toregexp                               0, 2
+0011 leave
+```
+
+In this case CRuby fetches the local variable with `getlocal_WC_1` as the second instruction to the "once" instruction sequence. When compiling CRuby, prism therefore will adjust the depth to account for this difference.
+
+## `rescue` clauses
+
+In CRuby, `rescue` clauses are implemented as their own instruction sequence, and therefore CRuby thinks the depth is one more than prism does. For example:
+
+```
+$ ruby --dump=insns -e 'begin; foo = 1; rescue; foo; end'
+== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,32)> (catch: true)
+== catch table
+| catch type: rescue st: 0000 ed: 0004 sp: 0000 cont: 0005
+| == disasm: #<ISeq:rescue in <main>@-e:1 (1,16)-(1,28)> (catch: true)
+| local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
+| [ 1] $!@0
+| 0000 getlocal_WC_0                          $!@0                      (   1)
+| 0002 putobject                              StandardError
+| 0004 checkmatch                             3
+| 0006 branchunless                           11
+| 0008 getlocal_WC_1                          foo@0[Li]
+| 0010 leave
+| 0011 getlocal_WC_0                          $!@0
+| 0013 throw                                  0
+| catch type: retry  st: 0004 ed: 0005 sp: 0000 cont: 0000
+|------------------------------------------------------------------------
+local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
+[ 1] foo@0
+0000 putobject_INT2FIX_1_                                             (   1)[Li]
+0001 dup
+0002 setlocal_WC_0                          foo@0
+0004 nop
+0005 leave
+```
+
+In the catch table, CRuby is reading the `foo` local variable using `getlocal_WC_1` as the fifth instruction to the "rescue" instruction sequence. When compiling CRuby, prism therefore will adjust the depth to account for this difference.
+
+Note that this includes the error reference, which can target local variables, as in:
+
+```
+$ ruby --dump=insns -e 'foo = 1; begin; rescue => foo; end'              
+== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,34)> (catch: true)
+== catch table
+| catch type: rescue st: 0003 ed: 0004 sp: 0000 cont: 0005
+| == disasm: #<ISeq:rescue in <main>@-e:1 (1,16)-(1,30)> (catch: true)
+| local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
+| [ 1] $!@0
+| 0000 getlocal_WC_0                          $!@0                      (   1)
+| 0002 putobject                              StandardError
+| 0004 checkmatch                             3
+| 0006 branchunless                           14
+| 0008 getlocal_WC_0                          $!@0
+| 0010 setlocal_WC_1                          foo@0
+| 0012 putnil
+| 0013 leave
+| 0014 getlocal_WC_0                          $!@0
+| 0016 throw                                  0
+| catch type: retry  st: 0004 ed: 0005 sp: 0000 cont: 0003
+|------------------------------------------------------------------------
+local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
+[ 1] foo@0
+0000 putobject_INT2FIX_1_                                             (   1)[Li]
+0001 setlocal_WC_0                          foo@0
+0003 putnil
+0004 nop
+0005 leave
+```
+
+Note that CRuby is writing to the `foo` local variable using the `setlocal_WC_1` instruction as the sixth instruction to the "rescue" instruction sequence. When compiling CRuby, prism therefore will adjust the depth to account for this difference.
+
+## Post execution blocks
+
+The `END {}` syntax allows executing code when the program exits. In CRuby, this is implemented as two nested instruction sequences. CRuby therefore thinks the depth is two more than prism does. For example:
+
+```
+$ ruby --dump=insns -e 'foo = 1; END { foo }'            
+== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,20)> (catch: false)
+local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
+[ 1] foo@0
+0000 putobject_INT2FIX_1_                                             (   1)[Li]
+0001 setlocal_WC_0                          foo@0
+0003 once                                   block in <main>, <is:0>
+0006 leave
+
+== disasm: #<ISeq:block in <main>@-e:0 (0,0)-(-1,-1)> (catch: false)
+0000 putspecialobject                       1                         (   1)
+0002 send                                   <calldata!mid:core#set_postexe, argc:0, FCALL>, block in <main>
+0005 leave
+
+== disasm: #<ISeq:block in <main>@-e:1 (1,9)-(1,20)> (catch: false)
+0000 getlocal                               foo@0, 2                  (   1)[LiBc]
+0003 leave                                  [Br]
+```
+
+In the instruction sequence corresponding to the code that gets executed inside the `END` block, CRuby is reading the `foo` local variable using `getlocal` as the second instruction to the `"block in <main>"` instruction sequence. When compiling CRuby, prism therefore will adjust the depth to account for this difference.

data/docs/ruby_api.md

@@ -25,6 +25,8 @@ The full API is documented below.
 * `Prism.load(source, serialized)` - load the serialized syntax tree using the source as a reference into a syntax tree
 * `Prism.parse_comments(source)` - parse the comments corresponding to the given source string and return them
 * `Prism.parse_file_comments(source)` - parse the comments corresponding to the given source file and return them
+* `Prism.parse_success?(source)` - parse the syntax tree corresponding to the given source string and return true if it was parsed without errors
+* `Prism.parse_file_success?(filepath)` - parse the syntax tree corresponding to the given source file and return true if it was parsed without errors
 
 ## Nodes
 

data/docs/serialization.md

@@ -9,24 +9,28 @@ The syntax tree still requires a copy of the original source, as for the most pa
 
 Let us define some simple types for readability.
 
-### varint
+### varuint
 
-A variable-length integer with the value fitting in `uint32_t` using between 1 and 5 bytes, using the [LEB128](https://en.wikipedia.org/wiki/LEB128) encoding.
+A variable-length unsigned integer with the value fitting in `uint32_t` using between 1 and 5 bytes, using the [LEB128](https://en.wikipedia.org/wiki/LEB128) encoding.
 This drastically cuts down on the size of the serialized string, especially when the source file is large.
 
+### varsint
+
+A variable-length signed integer with the value fitting in `int32_t` using between 1 and 5 bytes, using [ZigZag encoding](https://protobuf.dev/programming-guides/encoding/#signed-ints) into [LEB128].
+
 ### string
 
 | # bytes | field |
 | --- | --- |
-| varint | the length of the string in bytes |
+| varuint | the length of the string in bytes |
 | ... | the string bytes |
 
 ### location
 
 | # bytes | field |
 | --- | --- |
-| varint | byte offset into the source string where this location begins |
-| varint | length of the location in bytes in the source string |
+| varuint | byte offset into the source string where this location begins |
+| varuint | length of the location in bytes in the source string |
 
 ### comment
 
@@ -34,7 +38,6 @@ The comment type is one of:
 
 * 0=`INLINE` (`# comment`)
 * 1=`EMBEDDED_DOCUMENT` (`=begin`/`=end`)
-* 2=`__END__` (after `__END__`)
 
 | # bytes | field |
 | --- | --- |
@@ -72,17 +75,18 @@ The header is structured like the following table:
 | `1` | patch version number |
 | `1` | 1 indicates only semantics fields were serialized, 0 indicates all fields were serialized (including location fields) |
 | string | the encoding name |
-| varint | the start line |
-| varint | number of comments |
+| varsint | the start line |
+| varuint | number of comments |
 | comment* | comments |
-| varint | number of magic comments |
+| varuint | number of magic comments |
 | magic comment* | magic comments |
-| varint | number of errors |
+| location? | the optional location of the `__END__` keyword and its contents |
+| varuint | number of errors |
 | diagnostic* | errors |
-| varint | number of warnings |
+| varuint | number of warnings |
 | diagnostic* | warnings |
 | `4` | content pool offset |
-| varint | content pool size |
+| varuint | content pool size |
 
 After the header comes the body of the serialized string.
 The body consists of a sequence of nodes that is built using a prefix traversal order of the syntax tree.
@@ -103,6 +107,7 @@ Every field on the node is then appended to the serialized string. The fields ca
 * `constant?` - An optional variable-length integer that represents an index in the constant pool. If it's not present, then a single `0` byte will be written in its place.
 * `location` - A field that is a location. This is structured as a variable-length integer start followed by a variable-length integer length.
 * `location?` - A field that is a location that is optionally present. If the location is not present, then a single `0` byte will be written in its place. If it is present, then it will be structured just like the `location` child node.
+* `uint8` - A field that is an 8-bit unsigned integer. This is structured as a single byte.
 * `uint32` - A field that is a 32-bit unsigned integer. This is structured as a variable-length integer.
 
 After the syntax tree, the content pool is serialized. This is a list of constants that were referenced from within the tree. The content pool begins at the offset specified in the header. Constants can be either "owned" (in which case their contents are embedded in the serialization) or "shared" (in which case their contents represent a slice of the source string). The most significant bit of the constant indicates whether it is owned or shared.
@@ -159,7 +164,7 @@ serialize(const uint8_t *source, size_t length) {
 }
 ```
 
-The final argument to `pm_serialize_parse` is an optional string that controls the options to the parse function. This includes all of the normal options that could be passed to `pm_parser_init` through a `pm_options_t` struct, but serialized as a string to make it easier for callers through FFI. Note that no `varint` are used here to make it easier to produce the data for the caller, and also serialized size is less important here. The format of the data is structured as follows:
+The final argument to `pm_serialize_parse` is an optional string that controls the options to the parse function. This includes all of the normal options that could be passed to `pm_parser_init` through a `pm_options_t` struct, but serialized as a string to make it easier for callers through FFI. Note that no `varuint` are used here to make it easier to produce the data for the caller, and also serialized size is less important here. The format of the data is structured as follows:
 
 | # bytes | field                      |
 | ------- | -------------------------- |