prism 0.24.0 → 0.26.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 827194fd8b477f9f021677ca386e35fd8300daa86a09fc11fe66a85ee54da97d
-  data.tar.gz: 2a44a3cf5e31260a949965aaf2aa5fba0cd8b1ac910ae694d22fd8e42f4d6b9b
+  metadata.gz: abad576ba53ce985475eb5d69638557386eef255c8fe625728780ce8ff66844e
+  data.tar.gz: a3c5e24f5d1ee0fe04d282a3f40d643cd914a518985a9cadebbb53200e4d3003
 SHA512:
-  metadata.gz: 41dab6cd85f9f57bf2e2256e7b7fd7ac44c46883ad6d79c0056bff3b34500b2ecb28a1febeb1deed5a49b862c4f4382856ad2195639f86e770c006d1b6a41f8e
-  data.tar.gz: 0474d64c9bbd3385f65b00bf83e6b2d3c9543785c85b42d50bce59327e6af7a8fc13d54e5b485b6c930997dabe0fd81c43835243fc96fb28beb3523395ddde71
+  metadata.gz: d3255b2de64f3154500885ae2bd091389d6ac87c9b88cb8ff83b7d58026066c4e007cb7c486dcc8e8146132aad63a1c58ddd8ad598441d26bec7ad062409f6a9
+  data.tar.gz: 3d7563663f013a4dcc633b5e19d0a4605cae18bd0a0f99cc02c142198b25126f915185eef4080625c9a5867093a5bc5717db96ef8a8da6cdc69e849f65fbb7fd

data/BSDmakefile

@@ -0,0 +1,58 @@
+# GNU makefile proxy script for BSD make
+#
+# Written and maintained by Mahmoud Al-Qudsi <mqudsi@neosmart.net>
+# Copyright NeoSmart Technologies <https://neosmart.net/> 2014-2019
+# Obtain updates from <https://github.com/neosmart/gmake-proxy>
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# 1. Redistributions of source code must retain the above copyright notice, this
+# list of conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright notice,
+# this list of conditions and the following disclaimer in the documentation
+# and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+JARG =
+GMAKE = "gmake"
+# When gmake is called from another make instance, -w is automatically added
+# which causes extraneous messages about directory changes to be emitted.
+# Running with --no-print-directory silences these messages.
+GARGS = "--no-print-directory"
+
+.if "$(.MAKE.JOBS)" != ""
+    JARG = -j$(.MAKE.JOBS)
+.endif
+
+# bmake prefers out-of-source builds and tries to cd into ./obj (among others)
+# where possible. GNU Make doesn't, so override that value.
+.OBJDIR: ./
+
+# The GNU convention is to use the lowercased `prefix` variable/macro to
+# specify the installation directory. Humor them.
+GPREFIX = ""
+.if defined(PREFIX) && ! defined(prefix)
+    GPREFIX = 'prefix = "$(PREFIX)"'
+.endif
+
+.BEGIN: .SILENT
+	which $(GMAKE) || printf "Error: GNU Make is required!\n\n" 1>&2 && false
+
+.PHONY: FRC
+$(.TARGETS): FRC
+	$(GMAKE) $(GPREFIX) $(GARGS) $(.TARGETS:S,.DONE,,) $(JARG)
+
+.DONE .DEFAULT: .SILENT
+	$(GMAKE) $(GPREFIX) $(GARGS) $(.TARGETS:S,.DONE,,) $(JARG)

data/CHANGELOG.md

@@ -6,6 +6,72 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
 
 ## [Unreleased]
 
+## [0.26.0] - 2024-04-18
+
+### Added
+
+- Add `Prism::Node::fields`, which returns a list of `Prism::Reflection::Field` objects representing the fields of the node class. This is useful in metaprogramming contexts.
+- `Prism::Location#chop`, for removing the last byte from a location.
+- The void statement warning is now implemented.
+- The unreachable statement warning is now implemented.
+- A syntax error has been added for block arguments on yields, e.g., `yield(&foo)`.
+
+### Changed
+
+- Better fidelity to `parser` when translating heredocs with interpolation.
+- Fixed `RBI` and `RBS` types for `Prism::parse_*` signatures.
+- Remove some incorrect warnings about unused local variables.
+- More closely match CRuby error messages for global variables.
+- Fix an issue with `parser` translation when line continuations are found in string literals.
+
+## [0.25.0] - 2024-04-05
+
+### Added
+
+- `Prism::Translation::Ripper` is now able to mirror all of the Ripper APIs.
+- `Prism::Location#leading_comments` and `Prism::Location#trailing_comments` is added.
+- `Prism::Comment#slice` is added.
+- Warn for writing literal values in conditional predicates.
+- Check for `_POSIX_MAPPED_FILES` before using `mmap`.
+- `Prism::ItParametersNode` is added, to support `-> { it }`.
+- Parse integer and float literal values onto the tree.
+- Warn on duplicated hash keys and duplicated when clauses.
+- Ship much improved `RBI` and `RBS` types.
+- Support for the `-p`, `-n`, `-a`, and `-l` command line switches.
+- Warn on integer literals in flip-flops.
+- Support BSD make.
+- Add `Prism::WhenNode#then_keyword_loc`.
+- Support custom allocation functions through the `PRISM_XALLOCATOR` define.
+- Warn for certain keywrods at the end of the line.
+- Provide `pm_visit_node`, a C visitor API.
+- `Prism::parse_stream` is added, which works for any Ruby `IO` object.
+- Provide flags for regular expression literals for their derived encoding.
+- Provide flags for whether or not an interpolated string literal is frozen.
+- Add `Prism::StringNode.mutable?` for when a string is explicitly mutable, to support delineating chilled strings.
+- Warn for incorrect character literal syntax.
+- Warn for chained comparison operators.
+- Warn for `**` interpreted as an argument prefix.
+- Warn for `&` interpreted as an argument prefix.
+- `Prism::ShareableConstantNode` added to support ractors.
+- Warn for frozen string literals found after tokens.
+- Support `PRISM_BUILD_MINIMAL` to provide only the minimal necessary functionality to reduce the binary size.
+- Handle CLRF inside heredocs, strings, and regular expressions.
+- Mark inner strings in interpolated strings as frozen.
+- Support the `-x` command line switch.
+- Error messages now much more closely mirror CRuby.
+- Provide syntax errors for invalid block exits (`break`, `next`, `retry`, and `yield`).
+- Warn on unused local variables.
+- Do not syntax error on default parameter values that only write to the parameter.
+
+### Changed
+
+- Many improvements to the compatibility with the `whitequark/parser` translation.
+- Accept newlines before pattern terminators `)` or `]`.
+- `Prism::Node#start_offset` and `Prism::Node#end_offset` are now much more efficient.
+- Read files using `fread` instead of `mmap` when we're going to keep around the source through the Ruby API.
+- Fix `Sexp#line_max` setting in the `seattlerb/ruby_parser` translation layer.
+- Allow spaces before the encoding comment.
+
 ## [0.24.0] - 2024-02-15
 
 ### Added
@@ -392,7 +458,9 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
 
 - 🎉 Initial release! 🎉
 
-[unreleased]: https://github.com/ruby/prism/compare/v0.24.0...HEAD
+[unreleased]: https://github.com/ruby/prism/compare/v0.26.0...HEAD
+[0.26.0]: https://github.com/ruby/prism/compare/v0.25.0...v0.26.0
+[0.25.0]: https://github.com/ruby/prism/compare/v0.24.0...v0.25.0
 [0.24.0]: https://github.com/ruby/prism/compare/v0.23.0...v0.24.0
 [0.23.0]: https://github.com/ruby/prism/compare/v0.22.0...v0.23.0
 [0.22.0]: https://github.com/ruby/prism/compare/v0.21.0...v0.22.0

data/Makefile

@@ -6,17 +6,20 @@ Q1 = $(V:1=)
 Q = $(Q1:0=@)
 ECHO1 = $(V:1=@ :)
 ECHO = $(ECHO1:0=@ echo)
-FUZZ_OUTPUT_DIR = $(shell pwd)/fuzz/output
+FUZZ_OUTPUT_DIR = $(CURDIR)/fuzz/output
 
-SOEXT := $(shell ruby -e 'puts RbConfig::CONFIG["SOEXT"]')
+SOEXT ?= $(shell ruby -e 'puts RbConfig::CONFIG["SOEXT"]')
 
-CPPFLAGS := -Iinclude
-CFLAGS := -g -O2 -std=c99 -Wall -Werror -Wextra -Wpedantic -Wundef -Wconversion -Wno-missing-braces -fPIC -fvisibility=hidden
-CC := cc
+CPPFLAGS := -Iinclude $(CPPFLAGS)
+CFLAGS := -g -O2 -std=c99 -Wall -Werror -Wextra -Wpedantic -Wundef -Wconversion -Wno-missing-braces -fPIC -fvisibility=hidden $(CFLAGS)
+CC ?= cc
 WASI_SDK_PATH := /opt/wasi-sdk
 
-HEADERS := $(shell find include -name '*.h')
-SOURCES := $(shell find src -name '*.c')
+MAKEDIRS ?= mkdir -p
+RMALL ?= rm -f -r
+
+HEADERS := $(wildcard include/*.h include/*/*.h include/*/*/*.h')
+SOURCES := $(wildcard src/*.c src/*/*.c)
 SHARED_OBJECTS := $(subst src/,build/shared/,$(SOURCES:.c=.o))
 STATIC_OBJECTS := $(subst src/,build/static/,$(SOURCES:.c=.o))
 
@@ -45,17 +48,17 @@ java-wasm/src/test/resources/prism.wasm: Makefile $(SOURCES) $(HEADERS)
 
 build/shared/%.o: src/%.c Makefile $(HEADERS)
 	$(ECHO) "compiling $@"
-	$(Q) mkdir -p $(@D)
+	$(Q) $(MAKEDIRS) $(@D)
 	$(Q) $(CC) $(DEBUG_FLAGS) -DPRISM_EXPORT_SYMBOLS $(CPPFLAGS) $(CFLAGS) -c -o $@ $<
 
 build/static/%.o: src/%.c Makefile $(HEADERS)
 	$(ECHO) "compiling $@"
-	$(Q) mkdir -p $(@D)
+	$(Q) $(MAKEDIRS) $(@D)
 	$(Q) $(CC) $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) -c -o $@ $<
 
 build/fuzz.%: $(SOURCES) fuzz/%.c fuzz/fuzz.c
 	$(ECHO) "building $* fuzzer"
-	$(Q) mkdir -p $(@D)
+	$(Q) $(MAKEDIRS) $(@D)
 	$(ECHO) "building main fuzz binary"
 	$(Q) AFL_HARDEN=1 afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize-ignorelist=fuzz/asan.ignore -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@ $^
 	$(ECHO) "building cmplog binary"
@@ -66,7 +69,7 @@ build/fuzz.heisenbug.%: $(SOURCES) fuzz/%.c fuzz/heisenbug.c
 
 fuzz-debug:
 	$(ECHO) "entering debug shell"
-	$(Q) docker run -it --rm -e HISTFILE=/prism/fuzz/output/.bash_history -v $(shell pwd):/prism -v $(FUZZ_OUTPUT_DIR):/fuzz_output prism/fuzz
+	$(Q) docker run -it --rm -e HISTFILE=/prism/fuzz/output/.bash_history -v $(CURDIR):/prism -v $(FUZZ_OUTPUT_DIR):/fuzz_output prism/fuzz
 
 fuzz-docker-build: fuzz/docker/Dockerfile
 	$(ECHO) "building docker image"
@@ -76,17 +79,17 @@ fuzz-run-%: FORCE fuzz-docker-build
 	$(ECHO) "generating templates"
 	$(Q) bundle exec rake templates
 	$(ECHO) "running $* fuzzer"
-	$(Q) docker run --rm -v $(shell pwd):/prism prism/fuzz /bin/bash -c "FUZZ_FLAGS=\"$(FUZZ_FLAGS)\" make build/fuzz.$*"
+	$(Q) docker run --rm -v $(CURDIR):/prism prism/fuzz /bin/bash -c "FUZZ_FLAGS=\"$(FUZZ_FLAGS)\" make build/fuzz.$*"
 	$(ECHO) "starting AFL++ run"
-	$(Q) mkdir -p $(FUZZ_OUTPUT_DIR)/$*
-	$(Q) docker run -it --rm -v $(shell pwd):/prism -v $(FUZZ_OUTPUT_DIR):/fuzz_output prism/fuzz /bin/bash -c "./fuzz/$*.sh /fuzz_output/$*"
+	$(Q) $(MAKEDIRS) $(FUZZ_OUTPUT_DIR)/$*
+	$(Q) docker run -it --rm -v $(CURDIR):/prism -v $(FUZZ_OUTPUT_DIR):/fuzz_output prism/fuzz /bin/bash -c "./fuzz/$*.sh /fuzz_output/$*"
 FORCE:
 
 fuzz-clean:
-	$(Q) rm -f -r fuzz/output
+	$(Q) $(RMALL) fuzz/output
 
 clean:
-	$(Q) rm -f -r build
+	$(Q) $(RMALL) build
 
 .PHONY: clean fuzz-clean
 
@@ -94,6 +97,9 @@ all-no-debug: DEBUG_FLAGS := -DNDEBUG=1
 all-no-debug: OPTFLAGS := -O3
 all-no-debug: all
 
+minimal: CFLAGS := $(CFLAGS) -DPRISM_BUILD_MINIMAL
+minimal: all
+
 run: Makefile $(STATIC_OBJECTS) $(HEADERS) test.c
 	$(ECHO) "compiling test.c"
 	$(Q) $(CC) $(CPPFLAGS) $(CFLAGS) $(STATIC_OBJECTS) test.c

data/README.md

@@ -3,7 +3,7 @@
   <img alt="Prism Ruby parser" height="256px" src="https://github.com/ruby/prism/blob/main/doc/images/prism.png?raw=true">
 </div>
 
-This is a parser for the Ruby programming language. It is designed to be portable, error tolerant, and maintainable. It is written in C99 and has no dependencies. It is currently being integrated into [CRuby](https://github.com/ruby/ruby), [JRuby](https://github.com/jruby/jruby), [TruffleRuby](https://github.com/oracle/truffleruby), [Sorbet](https://github.com/sorbet/sorbet), and [Syntax Tree](https://github.com/ruby-syntax-tree/syntax_tree).
+This is a parser for the Ruby programming language. It is designed to be portable, error tolerant, and maintainable. It is written in C99 and has no dependencies.
 
 ## Overview
 
@@ -15,27 +15,33 @@ The repository contains the infrastructure for both a shared library (libprism)
 ├── Rakefile              configuration to compile the native extension and run the Ruby tests
 ├── bin
 │   ├── lex               runs the lexer on a file or string, prints the tokens, and compares to ripper
-│   └── parse             runs the parser on a file or string and prints the syntax tree
+│   ├── parse             runs the parse on a file or string and prints the AST
+│   └── prism             a CLI for development and debugging
 ├── config.yml            specification for tokens and nodes in the tree
-├── docs                  documentation about the project
+├── doc                   documentation website
+├── docs                  markdown documentation about the project
 ├── ext
 │   └── prism
 │       ├── extconf.rb    configuration to generate the Makefile for the native extension
 │       └── extension.c   the native extension that interacts with libprism
 ├── fuzz                  files related to fuzz testing
+├── gemfiles              gemfiles used by different Ruby versions in CI
 ├── include
 │   ├── prism             header files for the shared library
 │   └── prism.h           main header file for the shared library
 ├── java                  Java bindings for the shared library
+├── java-wasm             Java WASM bindings for the shared library
+├── javascript            JavaScript WASM bindings for the shared library
 ├── lib
 │   ├── prism             Ruby library files
 │   └── prism.rb          main entrypoint for the Ruby library
 ├── rakelib               various Rake tasks for the project
+├── rbi                   RBI type signatures for the Ruby library
 ├── rust
 │   ├── ruby-prism        Rustified crate for the shared library
 │   └── ruby-prism-sys    FFI binding for Rust
+├── sig                   RBS type signatures for the Ruby library
 ├── src
-│   ├── enc               various encoding files
 │   ├── util              various utility files
 │   └── prism.c           main entrypoint for the shared library
 ├── templates             contains ERB templates generated by templates/template.rb
@@ -51,7 +57,7 @@ The repository contains the infrastructure for both a shared library (libprism)
 To compile the shared library, you will need:
 
 * C99 compiler
-* make
+* GNU make
 * Ruby 2.7.0 or later
 
 Once you have these dependencies, run:
@@ -91,8 +97,41 @@ See the [CONTRIBUTING.md](CONTRIBUTING.md) file for more information. We additio
 * [Parser translation](docs/parser_translation.md)
 * [Parsing rules](docs/parsing_rules.md)
 * [Releasing](docs/releasing.md)
-* [Ripper](docs/ripper.md)
+* [Ripper translation](docs/ripper_translation.md)
 * [Ruby API](docs/ruby_api.md)
 * [RubyParser translation](docs/ruby_parser_translation.md)
 * [Serialization](docs/serialization.md)
 * [Testing](docs/testing.md)
+
+## Examples
+
+Prism has been integrated into the majority of Ruby runtimes, many libraries, and some applications. Below is a list of some of the projects that use Prism:
+
+### Runtimes
+
+* [CRuby](https://github.com/ruby/ruby/pull/7964) (via C)
+* [Garnet](https://github.com/camertron/garnet-js) (via WASM)
+* [JRuby](https://github.com/jruby/jruby/pull/8103) (via Java)
+* [Natalie](https://github.com/natalie-lang/natalie/pull/1213) (via C++ and Ruby)
+* [Opal](https://github.com/opal/opal/pull/2642) (via Ruby and WASM)
+* [TruffleRuby](https://github.com/oracle/truffleruby/issues/3117) (via Java)
+
+### Libraries
+
+* [dispersion](https://github.com/joeldrapper/dispersion)
+* [packwerk](https://github.com/Shopify/packwerk/pull/388) (via parser translator)
+* [rbi](https://github.com/Shopify/rbi)
+* [rails](https://github.com/rails/rails)
+  * [parsing renders](https://github.com/rails/rails/pull/49438)
+  * [parsing rdoc](https://github.com/rails/rails/pull/50870)
+  * [parsing tests](https://github.com/rails/rails/pull/51006)
+* [repl_type_completor](https://github.com/ruby/repl_type_completor)
+* [rubocop](https://docs.rubocop.org/rubocop/configuration.html#setting-the-parser-engine) (via parser translator)
+* [ruby-lsp](https://github.com/Shopify/ruby-lsp)
+* [smart_todo](https://github.com/Shopify/smart_todo/pull/69)
+* [sorbet-eraser](https://github.com/kddnewton/sorbet-eraser/pull/25)
+* [synvert](https://github.com/xinminlabs/synvert-core-ruby)
+
+### Applications
+
+* [gem.sh](https://github.com/marcoroth/gem.sh/pull/96)