prism 0.19.0 → 0.24.0

data/docs/releasing.md

@@ -8,23 +8,91 @@ To release a new version of Prism, perform the following steps:
   * Add a new section for the new version at the top of the file.
   * Fill in the relevant changes — it may be easiest to click the link for the `Unreleased` heading to find the commits.
   * Update the links at the bottom of the file.
-* Update the version in the following files:
-  * `prism.gemspec` in the `Gem::Specification#version=` method call
-  * `ext/prism/extension.h` in the `EXPECTED_PRISM_VERSION` macro
-  * `include/prism/version.h` in the version macros
-  * `javascript/package.json` in the `version` field
-  * `rust/prism-sys/tests/utils_tests.rs` in the `version_test` function
-  * `templates/java/org/prism/Loader.java.erb` in the `load` function
-  * `templates/javascript/src/deserialize.js.erb` in the version constants
-  * `templates/lib/prism/serialize.rb.erb` in the version constants
-* Run `bundle install` to update the `Gemfile.lock` file.
-* Update `rust/prism-sys/Cargo.toml` to match the new version and run `cargo build`
-* Update `rust/prism/Cargo.toml` to match the new version and run `cargo build`
-* Commit all of the updated files.
+* Update the version numbers in the various files that reference them:
+
+```sh
+export PRISM_MAJOR="x"
+export PRISM_MINOR="y"
+export PRISM_PATCH="z"
+export PRISM_VERSION="$PRISM_MAJOR.$PRISM_MINOR.$PRISM_PATCH"
+ruby -pi -e 'gsub(/spec\.version = ".+?"/, %Q{spec.version = "#{ENV["PRISM_VERSION"]}"})' prism.gemspec
+ruby -pi -e 'gsub(/EXPECTED_PRISM_VERSION ".+?"/, %Q{EXPECTED_PRISM_VERSION "#{ENV["PRISM_VERSION"]}"})' ext/prism/extension.h
+ruby -pi -e 'gsub(/PRISM_VERSION_MAJOR \d+/, %Q{PRISM_VERSION_MAJOR #{ENV["PRISM_MAJOR"]}})' include/prism/version.h
+ruby -pi -e 'gsub(/PRISM_VERSION_MINOR \d+/, %Q{PRISM_VERSION_MINOR #{ENV["PRISM_MINOR"]}})' include/prism/version.h
+ruby -pi -e 'gsub(/PRISM_VERSION_PATCH \d+/, %Q{PRISM_VERSION_PATCH #{ENV["PRISM_PATCH"]}})' include/prism/version.h
+ruby -pi -e 'gsub(/PRISM_VERSION ".+?"/, %Q{PRISM_VERSION "#{ENV["PRISM_VERSION"]}"})' include/prism/version.h
+ruby -pi -e 'gsub(/"version": ".+?"/, %Q{"version": "#{ENV["PRISM_VERSION"]}"})' javascript/package.json
+ruby -pi -e 'gsub(/lossy\(\), ".+?"/, %Q{lossy(), "#{ENV["PRISM_VERSION"]}"})' rust/ruby-prism-sys/tests/utils_tests.rs
+ruby -pi -e 'gsub(/\d+, "prism major/, %Q{#{ENV["PRISM_MAJOR"]}, "prism major})' templates/java/org/prism/Loader.java.erb
+ruby -pi -e 'gsub(/\d+, "prism minor/, %Q{#{ENV["PRISM_MINOR"]}, "prism minor})' templates/java/org/prism/Loader.java.erb
+ruby -pi -e 'gsub(/\d+, "prism patch/, %Q{#{ENV["PRISM_PATCH"]}, "prism patch})' templates/java/org/prism/Loader.java.erb
+ruby -pi -e 'gsub(/MAJOR_VERSION = \d+/, %Q{MAJOR_VERSION = #{ENV["PRISM_MAJOR"]}})' templates/javascript/src/deserialize.js.erb
+ruby -pi -e 'gsub(/MINOR_VERSION = \d+/, %Q{MINOR_VERSION = #{ENV["PRISM_MINOR"]}})' templates/javascript/src/deserialize.js.erb
+ruby -pi -e 'gsub(/PATCH_VERSION = \d+/, %Q{PATCH_VERSION = #{ENV["PRISM_PATCH"]}})' templates/javascript/src/deserialize.js.erb
+ruby -pi -e 'gsub(/MAJOR_VERSION = \d+/, %Q{MAJOR_VERSION = #{ENV["PRISM_MAJOR"]}})' templates/lib/prism/serialize.rb.erb
+ruby -pi -e 'gsub(/MINOR_VERSION = \d+/, %Q{MINOR_VERSION = #{ENV["PRISM_MINOR"]}})' templates/lib/prism/serialize.rb.erb
+ruby -pi -e 'gsub(/PATCH_VERSION = \d+/, %Q{PATCH_VERSION = #{ENV["PRISM_PATCH"]}})' templates/lib/prism/serialize.rb.erb
+ruby -pi -e 'gsub(/^version = ".+?"/, %Q{version = "#{ENV["PRISM_VERSION"]}"})' rust/ruby-prism-sys/Cargo.toml
+ruby -pi -e 'gsub(/^version = ".+?"/, %Q{version = "#{ENV["PRISM_VERSION"]}"})' rust/ruby-prism/Cargo.toml
+ruby -pi -e 'gsub(/^ruby-prism-sys = \{ version = ".+?"/, %Q{ruby-prism-sys = \{ version = "#{ENV["PRISM_VERSION"]}"})' rust/ruby-prism/Cargo.toml
+```
+
+* Update the `Gemfile.lock` file:
+
+```sh
+chruby ruby-3.4.0-dev
+bundle install
+```
+
+* Update the version-specific lockfiles:
+
+```sh
+chruby ruby-2.7.8 && BUNDLE_GEMFILE=gemfiles/2.7/Gemfile bundle install
+chruby ruby-3.0.6 && BUNDLE_GEMFILE=gemfiles/3.0/Gemfile bundle install
+chruby ruby-3.1.4 && BUNDLE_GEMFILE=gemfiles/3.1/Gemfile bundle install
+chruby ruby-3.2.3 && BUNDLE_GEMFILE=gemfiles/3.2/Gemfile bundle install
+chruby ruby-3.3.0 && BUNDLE_GEMFILE=gemfiles/3.3/Gemfile bundle install
+chruby ruby-3.4.0-dev && BUNDLE_GEMFILE=gemfiles/3.4/Gemfile bundle install
+chruby jruby-9.4.5.0 && BUNDLE_GEMFILE=gemfiles/jruby/Gemfile bundle install
+chruby truffleruby-23.1.2 && BUNDLE_GEMFILE=gemfiles/truffleruby/Gemfile bundle install
+chruby ruby-3.4.0-dev
+```
+
+* Update the cargo lockfiles:
+
+```sh
+bundle exec rake cargo:build
+```
+
+* Commit all of the updated files:
+
+```sh
+git commit -am "Bump to v$PRISM_VERSION"
+```
 
 ## Publishing
 
 * Update the GitHub release page with a copy of the latest entry in the `CHANGELOG.md` file.
-* Run `bundle exec rake release` to publish the gem to [rubygems.org](rubygems.org). Note that you must have access to the `prism` gem to do this.
-* Either download the `wasm` artifact from GitHub actions or generate it yourself with `make wasm`.
-* Run `npm publish` to publish the JavaScript package to [npmjs.com](npmjs.com). Note that you must have access to the `ruby-prism` package to do this.
+* Publish the gem to [rubygems.org](rubygems.org). Note that you must have access to the `prism` gem to do this.
+
+```sh
+bundle exec rake release
+```
+
+* Generate the `wasm` artifact (or download it from GitHub actions and put it in `javascript/src/prism.wasm`).
+
+```sh
+make wasm
+```
+
+* Publish the JavaScript package to [npmjs.com](npmjs.com). Note that you must have access to the `@ruby/prism` package to do this.
+
+```sh
+npm publish
+```
+
+* Publish the rust crate to [crates.io](crates.io). Note that you must have access to the `ruby-prism-sys` and `ruby-prism` crates to do this.
+
+```sh
+bundle exec rake cargo:publish:real
+```

data/docs/ruby_api.md

@@ -14,7 +14,7 @@ The full API is documented below.
 
 ## API
 
-* `Prism.dump(source, filepath)` - parse the syntax tree corresponding to the given source string and filepath, and serialize it to a string. Filepath can be nil.
+* `Prism.dump(source)` - parse the syntax tree corresponding to the given source string, and serialize it to a string
 * `Prism.dump_file(filepath)` - parse the syntax tree corresponding to the given source file and serialize it to a string
 * `Prism.lex(source)` - parse the tokens corresponding to the given source string and return them as an array within a parse result
 * `Prism.lex_file(filepath)` - parse the tokens corresponding to the given source file and return them as an array within a parse result

data/docs/ruby_parser_translation.md

@@ -0,0 +1,19 @@
+# ruby_parser translation
+
+Prism ships with the ability to translate its syntax tree into the syntax tree used by the [seattlerb/ruby_parser](https://github.com/seattlerb/ruby_parser) gem. This allows you to use tools built on top of the `ruby_parser` gem with the `prism` parser.
+
+## Usage
+
+You can call the `parse` and `parse_file` methods on the `Prism::Translation::RubyParser` module:
+
+```ruby
+filepath = "path/to/file.rb"
+Prism::Translation::RubyParser.parse_file(filepath)
+```
+
+This will return to you `Sexp` objects that mirror the result of calling `RubyParser` methods, as in:
+
+```ruby
+filepath = "path/to/file.rb"
+RubyParser.new.parse(File.read(filepath), filepath)
+```

data/docs/serialization.md

@@ -51,12 +51,21 @@ The comment type is one of:
 | location | the location of the key of the magic comment |
 | location | the location of the value of the magic comment |
 
-### diagnostic
+### error
 
 | # bytes | field |
 | --- | --- |
-| string | diagnostic message (ASCII-only characters) |
-| location | the location in the source this diagnostic applies to |
+| string | error message (ASCII-only characters) |
+| location | the location in the source this error applies to |
+| `1` | the level of the error: `0` for `fatal` |
+
+## warning
+
+| # bytes | field |
+| --- | --- |
+| string | warning message (ASCII-only characters) |
+| location | the location in the source this warning applies to |
+| `1` | the level of the warning: `0` for `default` and `1` for `verbose` |
 
 ## Structure
 
@@ -76,15 +85,17 @@ The header is structured like the following table:
 | `1` | 1 indicates only semantics fields were serialized, 0 indicates all fields were serialized (including location fields) |
 | string | the encoding name |
 | varsint | the start line |
+| varuint | number of newline offsets |
+| varuint* | newline offsets |
 | varuint | number of comments |
 | comment* | comments |
 | varuint | number of magic comments |
 | magic comment* | magic comments |
 | location? | the optional location of the `__END__` keyword and its contents |
 | varuint | number of errors |
-| diagnostic* | errors |
+| error* | errors |
 | varuint | number of warnings |
-| diagnostic* | warnings |
+| warning* | warnings |
 | `4` | content pool offset |
 | varuint | content pool size |
 
@@ -175,9 +186,12 @@ The final argument to `pm_serialize_parse` is an optional string that controls t
 | ...     | the encoding bytes         |
 | `1`     | frozen string literal      |
 | `1`     | suppress warnings          |
+| `1`     | syntax version, see [pm_options_version_t](https://github.com/ruby/prism/blob/main/include/prism/options.h) for valid values |
 | `4`     | the number of scopes       |
 | ...     | the scopes                 |
 
+Scopes are ordered from the outermost scope to the innermost one.
+
 Each scope is layed out as follows:
 
 | # bytes | field                      |