prism 0.17.0 → 0.18.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 69cdca044f91ad2a198666562fdffd6035323906da465743ebff2cbd34ccac5d
-  data.tar.gz: 3da1460885f7cabda4b1ae6438274ab64a6e966536587d98eab2b3c764d0b6c0
+  metadata.gz: ec5c0e5a9423ea40bad7c6ea608cb43dc0250c4d7119b4960271f547697540c1
+  data.tar.gz: d505901505b8c4eee195171721d68fe05dde76de171e5a80115a301ba030cc34
 SHA512:
-  metadata.gz: c01d8b62728fe1cbce99394d683c28b943b7962e6a1fc82d435b3190286612fc4d7e4aaab93a6bef7a75f8acdfa3b5a597acac3295e489a7dd89aada94b29b23
-  data.tar.gz: ce88826e2a46cb18fe5e89b1a7c18c8fba2387b9c9e5625ec9aac5e5bd1449b21f23624bd0a6b526c4a9fa156c679186f8b89c09d0a003d386663f4a6a33269e
+  metadata.gz: 3f3a4b92227885288d47e147f72b5cd2a27886222339ac08426a32ffe980b5b1397cf0356db93fcbdb2f6a3d16b1aa8db7185acbfe2a92473ef6adcf2aeda93e
+  data.tar.gz: 20528ed404ebdfb5e2093054cd7d1b28fed5f8d95c84df217eee9759ca25603cbd381943db985e17b8e4851756605aa1fb2fe9d41c0d050dfee9d770ffd915d7

data/CHANGELOG.md

@@ -6,6 +6,40 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
 
 ## [Unreleased]
 
+## [0.18.0] - 2023-11-21
+
+### Added
+
+- The `ParametersNode#signature` method is added, which returns the same thing as `Method#parameters`.
+- Visitor functionality has been added to the JavaScript API.
+- The `Node#to_dot` API has been added to convert syntax trees to Graphviz digraphs.
+- `IfNode` and `UnlessNode` now have a `then_keyword_loc` field.
+- Many more encodings are now supported.
+- Some new `Location` APIs have been added for dealing with characters instead of bytes, which are: `start_character_offset`, `end_character_offset`, `start_character_column`, and `end_character_column`.
+- A warning has been added for when `END {}` is used within a method.
+- `ConstantPathNode#full_name{,_parts}` will now raise an error if the receiver of the constant path is not itself a constant.
+- The `in` keyword and the `=>` operator now respect non-associativity.
+- The `..` and `...` operators now properly respect non-associativity.
+
+### Changed
+
+- Previously `...` in blocks was accepted, but it is now properly rejected.
+- **BREAKING**: `librubyparser.*` has been renamed to `libprism.*` in the C API.
+- We now properly reject floats with exponent and rational suffixes.
+- We now properly reject void value expressions.
+- **BREAKING**: The `--disable-static` option has been removed from the C extension.
+- The rescue modifier keyword is now properly parsed in terms of precedence.
+- We now properly reject defining a numbered parameter method.
+- **BREAKING**: `MatchWriteNode` now has a list of `targets`, which are local variable target nodes. This is instead of `locals` which was a constant list. This is to support writing to local variables outside the current scope. It has the added benefit of providing location information for the local variable targets.
+- **BREAKING**: `CaseNode` has been split into `CaseNode` and `CaseMatchNode`, the latter is used for `case ... in` expressions.
+- **BREAKING**: `StringConcatNode` has been removed in favor of using `InterpolatedStringNode` as a list.
+
+## [0.17.1] - 2023-11-03
+
+### Changed
+
+- Do not use constant nesting in RBI files.
+
 ## [0.17.0] - 2023-11-03
 
 ### Added
@@ -233,7 +267,9 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
 
 - 🎉 Initial release! 🎉
 
-[unreleased]: https://github.com/ruby/prism/compare/v0.17.0...HEAD
+[unreleased]: https://github.com/ruby/prism/compare/v0.18.0...HEAD
+[0.18.0]: https://github.com/ruby/prism/compare/v0.17.1...v0.18.0
+[0.17.1]: https://github.com/ruby/prism/compare/v0.17.0...v0.17.1
 [0.17.0]: https://github.com/ruby/prism/compare/v0.16.0...v0.17.0
 [0.16.0]: https://github.com/ruby/prism/compare/v0.15.1...v0.16.0
 [0.15.1]: https://github.com/ruby/prism/compare/v0.15.0...v0.15.1

data/Makefile

@@ -11,7 +11,7 @@ FUZZ_OUTPUT_DIR = $(shell pwd)/fuzz/output
 SOEXT := $(shell ruby -e 'puts RbConfig::CONFIG["SOEXT"]')
 
 CPPFLAGS := -Iinclude
-CFLAGS := -g -O2 -std=c99 -Wall -Werror -Wextra -Wpedantic -Wundef -Wconversion -fPIC -fvisibility=hidden
+CFLAGS := -g -O2 -std=c99 -Wall -Werror -Wextra -Wpedantic -Wundef -Wconversion -Wno-missing-braces -fPIC -fvisibility=hidden
 CC := cc
 WASI_SDK_PATH := /opt/wasi-sdk
 
@@ -22,15 +22,15 @@ STATIC_OBJECTS := $(subst src/,build/static/,$(SOURCES:.c=.o))
 
 all: shared static
 
-shared: build/librubyparser.$(SOEXT)
-static: build/librubyparser.a
+shared: build/libprism.$(SOEXT)
+static: build/libprism.a
 wasm: javascript/src/prism.wasm
 
-build/librubyparser.$(SOEXT): $(SHARED_OBJECTS)
+build/libprism.$(SOEXT): $(SHARED_OBJECTS)
 	$(ECHO) "linking $@"
 	$(Q) $(CC) $(DEBUG_FLAGS) $(CFLAGS) -shared -o $@ $(SHARED_OBJECTS)
 
-build/librubyparser.a: $(STATIC_OBJECTS)
+build/libprism.a: $(STATIC_OBJECTS)
 	$(ECHO) "building $@"
 	$(Q) $(AR) $(ARFLAGS) $@ $(STATIC_OBJECTS) $(Q1:0=>/dev/null)
 

data/README.md

@@ -7,7 +7,7 @@ This is a parser for the Ruby programming language. It is designed to be portabl
 
 ## Overview
 
-The repository contains the infrastructure for both a shared library (librubyparser) and a native CRuby extension. The shared library has no bindings to CRuby itself, and so can be used by other projects. The native CRuby extension links against `ruby.h`, and so is suitable in the context of CRuby.
+The repository contains the infrastructure for both a shared library (libprism) and a native CRuby extension. The shared library has no bindings to CRuby itself, and so can be used by other projects. The native CRuby extension links against `ruby.h`, and so is suitable in the context of CRuby.
 
 ```
 .
@@ -21,7 +21,7 @@ The repository contains the infrastructure for both a shared library (librubypar
 ├── ext
 │   └── prism
 │       ├── extconf.rb    configuration to generate the Makefile for the native extension
-│       └── extension.c   the native extension that interacts with librubyparser
+│       └── extension.c   the native extension that interacts with libprism
 ├── fuzz                  files related to fuzz testing
 ├── include
 │   ├── prism             header files for the shared library

data/config.yml

@@ -768,6 +768,26 @@ nodes:
 
           foo => [bar => baz]
                  ^^^^^^^^^^^^
+  - name: CaseMatchNode
+    fields:
+      - name: predicate
+        type: node?
+      - name: conditions
+        type: node[]
+      - name: consequent
+        type: node?
+        kind: ElseNode
+      - name: case_keyword_loc
+        type: location
+      - name: end_keyword_loc
+        type: location
+    comment: |
+      Represents the use of a case statement for pattern matching.
+
+          case true
+          in false
+          end
+          ^^^^^^^^^
   - name: CaseNode
     fields:
       - name: predicate
@@ -1385,6 +1405,8 @@ nodes:
         type: location?
       - name: predicate
         type: node
+      - name: then_keyword_loc
+        type: location?
       - name: statements
         type: node?
         kind: StatementsNode
@@ -1878,8 +1900,8 @@ nodes:
       - name: call
         type: node
         kind: CallNode
-      - name: locals
-        type: constant[]
+      - name: targets
+        type: node[]
     comment: |
       Represents writing local variables using a regular expression match with
       named capture groups.
@@ -2348,17 +2370,6 @@ nodes:
 
           foo; bar; baz
           ^^^^^^^^^^^^^
-  - name: StringConcatNode
-    fields:
-      - name: left
-        type: node
-      - name: right
-        type: node
-    comment: |
-      Represents the use of compile-time string concatenation.
-
-          "foo" "bar"
-          ^^^^^^^^^^^
   - name: StringNode
     fields:
       - name: flags
@@ -2446,6 +2457,8 @@ nodes:
         type: location
       - name: predicate
         type: node
+      - name: then_keyword_loc
+        type: location?
       - name: statements
         type: node?
         kind: StatementsNode

data/docs/build_system.md

@@ -16,14 +16,14 @@ The main solution for the second point seems a Makefile, otherwise many of the u
 ## General Design
 
 1. Templates are generated by `templates/template.rb`
-4. The `Makefile` compiles both `librubyparser.a` and `librubyparser.{so,dylib,dll}` from the `src/**/*.c` and `include/**/*.h` files
+4. The `Makefile` compiles both `libprism.a` and `libprism.{so,dylib,dll}` from the `src/**/*.c` and `include/**/*.h` files
 5. The `Rakefile` `:compile` task ensures the above prerequisites are done, then calls `make`,
-  and uses `Rake::ExtensionTask` to compile the C extension (using its `extconf.rb`), which uses `librubyparser.a`
+  and uses `Rake::ExtensionTask` to compile the C extension (using its `extconf.rb`), which uses `libprism.a`
 
 This way there is minimal duplication, and each layer builds on the previous one and has its own responsibilities.
 
 The static library exports no symbols, to avoid any conflict.
-The shared library exports some symbols, and this is fine since there should only be one librubyparser shared library
+The shared library exports some symbols, and this is fine since there should only be one libprism shared library
 loaded per process (i.e., at most one version of the prism *gem* loaded in a process, only the gem uses the shared library).
 
 ## The various ways to build prism
@@ -36,11 +36,11 @@ loaded per process (i.e., at most one version of the prism *gem* loaded in a pro
 
 The gem contains the pre-generated templates.
 When installing the gem, `extconf.rb` is used and that:
-* runs `make build/librubyparser.a`
+* runs `make build/libprism.a`
 * compiles the C extension with mkmf
 
 When installing the gem on JRuby and TruffleRuby, no C extension is built, so instead of the last step,
-there is Ruby code using FFI which uses `librubyparser.{so,dylib,dll}`
+there is Ruby code using FFI which uses `libprism.{so,dylib,dll}`
 to implement the same methods as the C extension, but using serialization instead of many native calls/accesses
 (JRuby does not support C extensions, serialization is faster on TruffleRuby than the C extension).
 
@@ -67,7 +67,7 @@ The script generates the templates when importing.
 Then when `mx build` builds TruffleRuby and the `prism` mx project inside, it runs `make`.
 
 Then the `prism bindings` mx project is built, which contains the [bindings](https://github.com/oracle/truffleruby/blob/master/src/main/c/prism_bindings/src/prism_bindings.c)
-and links to `librubyparser.a` (to avoid exporting symbols, so no conflict when installing the prism gem).
+and links to `libprism.a` (to avoid exporting symbols, so no conflict when installing the prism gem).
 
 ### Building prism as part of JRuby
 

data/docs/building.md

@@ -10,7 +10,7 @@ All of the source files match `src/**/*.c` and all of the headers match `include
 The following flags should be used to compile prism:
 
 * `-std=c99` - Use the C99 standard
-* `-Wall -Wconversion -Wextra -Wpedantic -Wundef` - Enable the warnings we care about
+* `-Wall -Wconversion -Wextra -Wpedantic -Wundef -Wno-missing-braces` - Enable the warnings we care about
 * `-Werror` - Treat warnings as errors
 * `-fvisibility=hidden` - Hide all symbols by default
 

data/docs/configuration.md

@@ -11,6 +11,7 @@ A lot of code in prism's repository is templated from a single configuration fil
 * `java/org/prism/Nodes.java` - for defining the nodes in Java
 * `lib/prism/compiler.rb` - for defining the compiler for the nodes in Ruby
 * `lib/prism/dispatcher.rb` - for defining the dispatch visitors for the nodes in Ruby
+* `lib/prism/dot_visitor.rb` - for defining the dot visitor for the nodes in Ruby
 * `lib/prism/dsl.rb` - for defining the DSL for the nodes in Ruby
 * `lib/prism/mutation_compiler.rb` - for defining the mutation compiler for the nodes in Ruby
 * `lib/prism/node.rb` - for defining the nodes in Ruby

data/docs/encoding.md

@@ -12,43 +12,79 @@ If the file is not encoded in UTF-8, the user must specify the encoding in a "ma
 
 The key of the comment can be either "encoding" or "coding". The value of the comment must be a string that is a valid encoding name. The encodings that prism supports by default are:
 
-* `ascii`
-* `ascii-8bit`
-* `big5`
-* `binary`
-* `cp932`
-* `euc-jp`
-* `gbk`
-* `iso-8859-1`
-* `iso-8859-2`
-* `iso-8859-3`
-* `iso-8859-4`
-* `iso-8859-5`
-* `iso-8859-6`
-* `iso-8859-7`
-* `iso-8859-8`
-* `iso-8859-9`
-* `iso-8859-10`
-* `iso-8859-11`
-* `iso-8859-13`
-* `iso-8859-14`
-* `iso-8859-15`
-* `iso-8859-16`
-* `koi8-r`
-* `shift_jis`
-* `sjis`
-* `us-ascii`
-* `utf-8`
-* `utf8-mac`
-* `windows-31j`
-* `windows-1251`
-* `windows-1252`
+* `ASCII-8BIT`
+* `Big5`
+* `Big5-HKSCS`
+* `Big5-UAO`
+* `CP51932`
+* `CP850`
+* `CP852`
+* `CP855`
+* `EUC-JP`
+* `GB1988`
+* `GBK`
+* `IBM437`
+* `IBM720`
+* `IBM737`
+* `IBM775`
+* `IBM852`
+* `IBM855`
+* `IBM857`
+* `IBM860`
+* `IBM861`
+* `IBM862`
+* `IBM864`
+* `IBM865`
+* `IBM866`
+* `IBM869`
+* `ISO-8859-1`
+* `ISO-8859-2`
+* `ISO-8859-3`
+* `ISO-8859-4`
+* `ISO-8859-5`
+* `ISO-8859-6`
+* `ISO-8859-7`
+* `ISO-8859-8`
+* `ISO-8859-9`
+* `ISO-8859-10`
+* `ISO-8859-11`
+* `ISO-8859-13`
+* `ISO-8859-14`
+* `ISO-8859-15`
+* `ISO-8859-16`
+* `KOI8-R`
+* `macCentEuro`
+* `macCroatian`
+* `macCyrillic`
+* `macGreek`
+* `macIceland`
+* `macRoman`
+* `macRomania`
+* `macThai`
+* `macTurkish`
+* `macUkraine`
+* `Shift_JIS`
+* `TIS-620`
+* `US-ASCII`
+* `UTF-8`
+* `UTF8-MAC`
+* `Windows-1250`
+* `Windows-1251`
+* `Windows-1252`
+* `Windows-1253`
+* `Windows-1254`
+* `Windows-1255`
+* `Windows-1256`
+* `Windows-1257`
+* `Windows-1258`
+* `Windows-31J`
+* `Windows-874`
 
 For each of these encodings, prism provides a function for checking if the subsequent bytes form an alphabetic or alphanumeric character.
 
 ## Support for other encodings
 
-If an encoding is encountered that is not supported by prism, prism will call a user-provided callback function with the name of the encoding if one is provided. That function can be registered with `pm_parser_register_encoding_decode_callback`. The user-provided callback function can then provide a pointer to an encoding struct that contains the requisite functions that prism will use those to parse identifiers going forward.
+If an encoding is encountered that is not supported by prism, prism will call a user-provided callback function with the name of the encoding if one is provided. That function can be registered with `pm_parser_register_encoding_decode_callback`. The user-provided callback function can then provide a pointer to an encoding struct that contains the requisite functions that prism will use to parse identifiers going forward.
 
 If the user-provided callback function returns `NULL` (the value also provided by the default implementation in case a callback was not registered), an error will be added to the parser's error list and parsing will continue on using the default UTF-8 encoding.
 

data/docs/heredocs.md

@@ -12,7 +12,7 @@ When a heredoc identifier is encountered in the regular process of lexing, we pu
 
 We also set the special `parser.next_start` field which is a pointer to the place in the source where we should start lexing the next token. This is set to the pointer of the character immediately following the next newline.
 
-Note that if the `parser.heredoc_end` field is already set, then it means we have already encountered a heredoc on this line. In that case the `parser.next_start` field will be set to the `parser.heredoc_end` field. This is because we want to skip past the heredoc previous heredocs on this line and instead lex the body of this heredoc.
+Note that if the `parser.heredoc_end` field is already set, then it means we have already encountered a heredoc on this line. In that case the `parser.next_start` field will be set to the `parser.heredoc_end` field. This is because we want to skip past the previous heredocs on this line and instead lex the body of this heredoc.
 
 ## 2. Lexing the body
 

data/docs/javascript.md

@@ -24,7 +24,7 @@ const parse = await loadPrism();
 
 ## Browser
 
-To use the package from the browser, you will need to do some additional work. The [javascript/example.html](javascript/example.html) file shows an example of running Prism in the browser. You will need to instantiate the WebAssembly module yourself and then pass it to the `parsePrism` function.
+To use the package from the browser, you will need to do some additional work. The [javascript/example.html](../javascript/example.html) file shows an example of running Prism in the browser. You will need to instantiate the WebAssembly module yourself and then pass it to the `parsePrism` function.
 
 First, get a shim for WASI since not all browsers support it yet.
 
@@ -74,6 +74,34 @@ A ParseResult object is very similar to the Prism::ParseResult object from Ruby.
 console.log(JSON.stringify(parseResult.value, null, 2));
 ```
 
+## Visitors
+
+Prism allows you to traverse the AST of parsed Ruby code using visitors.
+
+Here's an example of a custom `FooCalls` visitor:
+
+```js
+import { loadPrism, Visitor } from "@ruby/prism"
+
+const parse = await loadPrism();
+const parseResult = parse("foo()");
+
+class FooCalls extends Visitor {
+  visitCallNode(node) {
+    if (node.name === "foo") {
+      // Do something with the node
+    }
+
+    // Call super so that the visitor continues walking the tree
+    super.visitCallNode(node);
+  }
+}
+
+const fooVisitor = new FooCalls();
+
+parseResult.value.accept(fooVisitor);
+```
+
 ## Building
 
 To build the WASM package yourself, first obtain a copy of `wasi-sdk`. You can retrieve this here: <https://github.com/WebAssembly/wasi-sdk>. Next, run:

data/docs/releasing.md

@@ -4,7 +4,7 @@ To release a new version of Prism, perform the following steps:
 
 ## Preparation
 
-* Update the CHANGELOG.md file.
+* Update the `CHANGELOG.md` file.
   * Add a new section for the new version at the top of the file.
   * Fill in the relevant changes — it may be easiest to click the link for the `Unreleased` heading to find the commits.
   * Update the links at the bottom of the file.
@@ -24,4 +24,7 @@ To release a new version of Prism, perform the following steps:
 
 ## Publishing
 
+* Update the GitHub release page with a copy of the latest entry in the `CHANGELOG.md` file.
 * Run `bundle exec rake release` to publish the gem to [rubygems.org](rubygems.org). Note that you must have access to the `prism` gem to do this.
+* Either download the `wasm` artifact from GitHub actions or generate it yourself with `make wasm`.
+* Run `npm publish` to publish the JavaScript package to [npmjs.com](npmjs.com). Note that you must have access to the `ruby-prism` package to do this.

data/docs/ruby_api.md

@@ -25,3 +25,17 @@ The full API is documented below.
 * `Prism.load(source, serialized)` - load the serialized syntax tree using the source as a reference into a syntax tree
 * `Prism.parse_comments(source)` - parse the comments corresponding to the given source string and return them
 * `Prism.parse_file_comments(source)` - parse the comments corresponding to the given source file and return them
+
+## Nodes
+
+Once you have nodes in hand coming out of a parse result, there are a number of common APIs that are available on each instance. They are:
+
+* `#accept(visitor)` - a method that will immediately call `visit_*` to specialize for the node type
+* `#child_nodes` - a positional array of the child nodes of the node, with `nil` values for any missing children
+* `#compact_child_nodes` - a positional array of the child nodes of the node with no `nil` values
+* `#copy(**keys)` - a method that allows creating a shallow copy of the node with the given keys overridden
+* `#deconstruct`/`#deconstruct_keys(keys)` - the pattern matching interface for nodes
+* `#inspect` - a string representation that looks like the syntax tree of the node
+* `#location` - a `Location` object that describes the location of the node in the source file
+* `#to_dot` - convert the node's syntax tree into graphviz dot notation
+* `#type` - a symbol that represents the type of the node, useful for quick comparisons