codemirror-lang-raku 0.3.0 → 0.4.1

package/CHANGELOG.md

@@ -1,4 +1,70 @@
-## 0.3.0 (2026-10-07)
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.4.1] - 2026-11-10
+
+### Internal
+
+- Releases are prepared by [tagpr](https://github.com/Songmu/tagpr): a
+  release pull request carries the version bump and this file's section,
+  and merging it tags the release and publishes it to npm with
+  [Trusted Publishing](https://docs.npmjs.com/trusted-publishers), with
+  provenance. The text of each GitHub release is still this file's section
+  for the version.
+
+## [0.4.0] - 2026-10-07
+
+Most of this was found by parsing the test suite of
+[mutsu](https://github.com/tokuhirom/mutsu), an implementation of Raku.
+
+### Breaking changes
+
+- The name of a constant with a sigil, as in `constant $LIMIT = 3`, is a
+  `ConstantName` inside the `ConstantDeclaration`. It was a
+  `VariableName` after the declaration, which had no name.
+- A `:sym<…>` that follows the name of a declaration after blanks, as in
+  `token word :sym<long> { }`, is part of the `RegexName`.
+- A declarator comment with brackets, `#|( … )` or `#=( … )`, is one
+  `DocComment` up to its closing bracket. It ended with its first line,
+  and still does when the bracket is not closed.
+
+### New features
+
+- Corner quotes (`｢…｣`), also after a quote word (`Q｢…｣`), the low
+  curly quotes (`„…”`, `„…“`, `‚…’`, `‚…‘`) and the reversed `”…”`.
+- A quote word takes a backtick or a symbol outside of ASCII for a
+  delimiter: ``qx`…` ``, `Q♥…♥`.
+- Fractions that are one character (`½`, `⅔`) are numbers, and a power
+  in superscript (`$x²`) is an operator.
+- Symbols outside of ASCII are operators: the atomic operators
+  (`⚛+=`), and the operators of one's own (`1 ⚡ 2`).
+- The set operators that are written in parentheses, as in
+  `$a (<=) $b` and `$x (elem) $s`, are operators.
+- An operator in brackets as a routine, as in `&[+]`, is a
+  `VariableName`.
+- The name of an operator can be written in double angles:
+  `infix:<< plus-one >>`, `&infix:<<(>=)>>`.
+- The variable `$¢`.
+
+### Bug fixes
+
+- In a regex, `‘…’`, `“…”`, `‚…’`, `„…”` and `｢…｣` are quotes, in which
+  the delimiter of the regex does not end it: `m/ab ‘/’ c/`.
+- A list of words in a regex, `/ < a ' b > /`, no longer starts a quote
+  at a `'` in it.
+- A regex does not end at its delimiter in code: in the arguments of an
+  assertion (`<name: /a/ >`, `<:name(/a/)>`), in `$( … )`, and in a
+  declaration (`:my $m = $/;`).
+- A `$/` in the replacement of a substitution does not end it:
+  `s/(a)/[$/]/`.
+
+## [0.3.0] - 2026-10-07
 
 ### Breaking changes
 
@@ -18,7 +84,7 @@
   quote that interpolates and spans several lines followed the opener
   of the heredoc on its line.
 
-## 0.2.0 (2026-10-07)
+## [0.2.0] - 2026-10-07
 
 ### New features
 
@@ -84,7 +150,7 @@
   longer starts a heredoc. Heredocs are now found from the quote tokens
   of the line, not from its text.
 
-## 0.1.0 (2026-10-06)
+## [0.1.0] - 2026-10-06
 
 ### New features
 
@@ -100,3 +166,9 @@ First release.
   of Pod blocks, heredocs and embedded comments
 - Completion of keywords and of the commonly used built-in types,
   routines, methods and special variables
+
+[0.4.1]: https://github.com/kan/codemirror-lang-raku/releases/tag/v0.4.1
+[0.4.0]: https://github.com/kan/codemirror-lang-raku/releases/tag/v0.4.0
+[0.3.0]: https://github.com/kan/codemirror-lang-raku/releases/tag/v0.3.0
+[0.2.0]: https://github.com/kan/codemirror-lang-raku/releases/tag/v0.2.0
+[0.1.0]: https://github.com/kan/codemirror-lang-raku/releases/tag/v0.1.0

package/README.md

@@ -9,10 +9,11 @@
 indentation, code folding, bracket matching and completion, built on a
 [Lezer](https://lezer.codemirror.net/) grammar.
 
-It covers comments and Pod, quotes of all kinds (`'…'`, `"…"`, `q` / `qq` /
-`Q` with any delimiter, heredocs, `<word lists>`) with interpolation,
-regexes, numbers, variables with sigils and twigils, keywords, operators,
-blocks and declarations.
+It covers comments and Pod, quotes of all kinds (`'…'`, `"…"`, `｢…｣`,
+curly quotes, `q` / `qq` / `Q` with any delimiter, heredocs,
+`<word lists>`) with interpolation, regexes and grammars, numbers,
+variables with sigils and twigils, keywords, operators, including the
+ones written in Unicode, blocks and declarations.
 
 Try it in the [demo](https://kan.github.io/codemirror-lang-raku/), which
 shows the syntax tree next to the editor.
@@ -87,33 +88,48 @@ on incomplete code.
 |---|---|
 | `PackageDeclaration` | `class`, `role`, `grammar`, `module`, `package`, with a `PackageName` child, up to the body `Block` or the `;` of a `unit` declaration |
 | `RoutineDeclaration` | `sub`, `method`, `submethod`, with a `RoutineName` child (absent for anonymous routines), up to the body `Block`. Also the `foo(…) { }` of `multi foo(…) { }`, which declares a sub without the word `sub`: that node starts with its `RoutineName` |
-| `RegexDeclaration` | `token`, `rule`, `regex`, with a `RegexName` child, up to the body `Block`, which holds one `Regex` |
-| `EnumDeclaration`, `SubsetDeclaration`, `ConstantDeclaration` | The declarator and its `EnumName`, `SubsetName` or `ConstantName` |
+| `RegexDeclaration` | `token`, `rule`, `regex`, with a `RegexName` child, up to the body `Block`, which holds one `Regex`. The name includes a `:sym<…>`, with the blanks before it if there are any (`word :sym<long>`) |
+| `EnumDeclaration`, `SubsetDeclaration`, `ConstantDeclaration` | The declarator and its `EnumName`, `SubsetName` or `ConstantName`. The name of a constant includes its sigil if it has one (`constant $LIMIT`). A constant without a sigil that is declared with a backslash (`constant \x`) has no `ConstantName` |
 | `Block`, `Parens`, `Brackets` | `{ }`, `( )`, `[ ]` |
-| `VariableName`, `AttributeName`, `SpecialVariable` | `$x` / `$^a`, `$!x` / `$.x`, `$*x` / `$?x` |
+| `VariableName`, `AttributeName`, `SpecialVariable` | `$x` / `$^a`, `$!x` / `$.x`, `$*x` / `$?x`. An operator that is referred to as a routine is one `VariableName`: `&infix:<+>`, `&[+]` |
+| `Identifier`, `TypeName` | A bare name (`say`, `my-sub`), and a capitalized one (`Int`, `Foo::Bar`), with its smiley if it has one (`Int:D`) |
+| `MethodCall` | The `.` and the `MethodName` of `.name`, `.?name` and the like. The arguments are the `Parens` after it |
+| `PairKey` | `:key`, `:!key`. The value is the node after it |
+| `Number`, `Version` | `42`, `1.5e3`, `0x1F`, `:16<FF>`, `½`, and `v6.d` |
+| `True`, `False`, `Nil` | These three words |
+| `Operator` | An operator made of symbols, a word operator where an operator is expected (`x`, `min`, `Z+`), a set operator in parentheses (`(elem)`) |
 | `StringLiteral` | A quote of any kind. One that interpolates has `Escape`, `Interpolation`, variable, `Brackets`, `Parens` and `MethodCall` children, and a `NestedDelimiters` for each pair of its own delimiters in it (the `[b]` of `qq[a [b] c]`) |
 | `Heredoc` | The text of a heredoc, from the line after its `q:to/END/` opener through its terminator. The opener is a `StringLiteral`. The text of a `qq:to` heredoc has the same children as a string that interpolates. The texts of the heredocs that one line opens are a single node |
-| `Regex` | `/…/`, `rx//`, `m//`, `s///`, `tr///`, or the body of a regex declaration. Its children are `CharacterClass` (`<[a..z]>`), `Assertion` (`<name>`, `<?before …>`), `StringLiteral`, `Escape` (`\d`), `Operator` (quantifiers, `\|`, anchors), `VariableName` (`$x`, `$<name>`), `LineComment`, and a `Block` for each `{ }` of code |
-| `LineComment`, `DocComment`, `BlockComment` | `#`, `#|` / `#=`, `` #`( ) `` |
+| `Regex` | `/…/`, `rx//`, `m//`, `s///`, `tr///`, or the body of a regex declaration. Its children are `CharacterClass` (`<[a..z]>`), `Assertion` (`<name>`, `<?before …>`), `StringLiteral`, `Escape` (`\d`), `Operator` (quantifiers, `\|`, anchors), `VariableName` (`$x`, `$<name>`), `LineComment`, `BlockComment`, and a `Block` for each `{ }` of code |
+| `LineComment`, `DocComment`, `BlockComment` | `#`, `#\|` / `#=`, `` #`( ) ``. A declarator comment with brackets, `#\|( … )`, is one `DocComment` up to its closing bracket |
 | `Pod` | `=begin` … `=end` and the other Pod blocks. Its children are `PodDirective` (`=begin`, `=head1`, `=end pod`), `PodHeading` (the text after `=head1`), and the formatting codes `PodStrong` (`B<…>`), `PodEmphasis` (`I<…>`), `PodCode` (`C<…>`), `PodLink` (`L<…>`) and `PodFormat` (the others) |
 
-Each declarator keyword is a child node named after the keyword (`class`,
-`sub`, `token`, …). The words `unit`, `multi`, `proto`, `only`, `my` and
+Each keyword is a node named after the keyword (`if`, `for`, `my`,
+`is`, `eq`, …), and so is each phaser (`BEGIN`, `LEAVE`, …). A
+declarator keyword is a child of its declaration node (`class`, `sub`,
+`token`, …). The words `unit`, `multi`, `proto`, `only`, `my` and
 `our` are not part of the declaration node. They are the nodes right
 before it, also in `multi foo(…) { }`, where the declaration node has no
 declarator keyword.
 
+Statements and expressions have no nodes: `if $x { }` is the nodes `if`,
+`VariableName` and `Block` next to each other.
+
 ## Known limitations
 
 Not supported yet:
 
 - Nothing is highlighted in the replacement of `s/…/…/`, nor in
-  `tr/…/…/`.
+  `tr/…/…/`. The second pair of brackets of `tr[…][…]` is read as code.
+  A replacement that is a block of code with the delimiter in it, as in
+  `s/x/{ "a/b" }/`, ends at that delimiter.
 - In a regex, a `<…>` is one
   token, with the code or the nested `<…>` in it, and has to be closed on
   its line unless it is a character class. Groups (`[ ]`, `( )`) and
   adverbs (`:i`, `:my`) are plain text, so the text after `:my` is read
-  as regex.
+  as regex, and so is the code in `$( … )`. Such code does not end the
+  regex when it holds the delimiter (`/ :my $m = $/; /`), if it is on
+  one line.
 - A heredoc is interpolated when everything interpolates in it, as in
   `qq:to`. One that interpolates in part (`q:c:to`, `qq:!s:to`) is not
   highlighted inside. When a line opens several heredocs, they are
@@ -129,9 +145,13 @@ Not supported yet:
   to 10 lines. A longer one is not taken to be a word list. Nothing is
   interpolated in a list that holds an unbalanced `{` or its own
   delimiter: it is a single token.
-- A Pod block is read where a statement or a term can stand, not
-  everywhere that whitespace can: one between `class` and the name of
-  the class is not read as Pod.
+- A Pod block between a declarator and its name, as between `class` and
+  the name of the class, hides the name: the declaration then has no
+  `PackageName`. A comment there does not.
+- In a quote that has braces for delimiters, a `{ }` is taken to be a
+  nested pair of the delimiters, not a block of code: the `{$b + 1}` of
+  `qq{a {$b + 1} c}` is text, in which `$b` is interpolated. With other
+  delimiters, as in `qq[a {$b + 1} c]`, it is a block.
 - A quote adverb is switched off by the argument `False` or `0`, as in
   `qq:c(False)[…]`. With any other argument, such as a variable, it
   counts as switched on.
@@ -139,8 +159,14 @@ Not supported yet:
   command. Text that unbalances every bracket that is tried gets
   `` #`( `` … `)`, which then does not cover exactly that text.
 - A user-defined operator is recognized where it is declared
-  (`sub infix:<+++>`) and as a routine (`&infix:<+++>`), not where it is
-  used as an operator.
+  (`sub infix:<+++>`) and as a routine (`&infix:<+++>`). Where it is
+  used, it is an operator when it is made of operator characters or of
+  symbols outside of ASCII (`1 ⚡ 2`). One that is a word is a plain
+  name, also between hyper marks (`@a »plus« @b`). A term or a circumfix of one's own is not known: one that is
+  a symbol is highlighted as an operator, and one that is a letter
+  outside of the ranges for names, such as `ℵ`, is an error.
+- The `native` declarator, which NativeCall uses internally, does not
+  make a declaration node.
 - Completion does not know types. After a `.` it offers the built-in
   methods and every method that the document declares, whatever the
   invocant is. Names from other files are not offered.
@@ -148,7 +174,8 @@ Not supported yet:
   `has`, in a signature, after the `->` of a pointy block, or by being a
   placeholder variable (`$^a`, `$:a`). A sigilless variable is offered
   when it is declared with a backslash (`my \x`, `sub f(\x)`, `-> \x`),
-  not as a term (`my constant x`, `sub term:<x>`).
+  and a constant when it is declared with `constant`. A term that is
+  declared as a routine (`sub term:<x>`) is not offered.
 - In a string, only the code in `{ }` and in a subscript is completed,
   not the method name of `"$x.name()"`.
 
@@ -164,6 +191,16 @@ Heuristics that can be wrong:
   the line before it does not end in a `;` or a `}`. Inside `( )` and
   `[ ]`, indentation follows the brackets only.
 - A capitalized word is taken to be a type name.
+- A symbol outside of ASCII is taken to be an operator, whatever it was
+  declared as.
+- A set operator in parentheses, such as `(elem)` or `(<=)`, is an
+  operator after a term and a space: `$a (elem) $b`. Right after the
+  term, the parentheses hold arguments: `$f(elem)`.
+- An operator in brackets, `&[+]`, is a routine when the operator is at
+  most 11 characters long, and not after a term, where the `&` is an
+  operator.
+- An operator name in double angles, as in `infix:<< plus >>`, has to
+  end within 20 characters of where it starts.
 - Whether `/` starts a regex and `<` starts a word list is decided from
   the token before it. After a bare name, spacing decides: `say /x/` and
   `say <a b>` are a regex and a word list, `foo / 2` and `n < 3` are not.
@@ -172,7 +209,8 @@ Heuristics that can be wrong:
   routine of your own has one of these names. After a closing brace, a
   line break decides.
 - A `<…>` right after a term is a subscript only when it holds plain words
-  on one line: `%h<key>`, but not `$a<$b`. A `<<…>>` or `«…»` that touches
+  on one line: `%h<key>`, but not `$a<$b`, nor a key with a quote or
+  other punctuation in it (`%h<a"b>`). A `<<…>>` or `«…»` that touches
   the term before it is a subscript when it is at most 24 characters
   long and starts and ends in a word, a variable or a quote: `%h«$key»`,
   but not the hyper operators of `@a<<+>>@b` and `@a«R-»@b`. A hyper
@@ -183,10 +221,14 @@ Heuristics that can be wrong:
   between. One that is being typed takes the code up to there for its
   content.
 - In a `<<…>>` word list, a `>>` in a subscript or a call that is
-  interpolated ends the list: `<<@a[1 >> 2]>>`. A heredoc that is opened
-  in a block in a word list or in a regex literal is not read as one.
+  interpolated ends the list: `<<@a[1 >> 2]>>`, and so does the `>` of
+  a subscript right before the end, as in `<<x %h<b>>>`. A heredoc that
+  is opened in a block in a word list or in a regex literal is not read
+  as one.
 - `q`, `qq`, `Q`, `m`, `rx`, `s`, `tr` and their variants start a quote
-  when a delimiter follows directly, so a sigilless `s/2` is misread.
+  when a delimiter follows directly: a bracket, one of
+  ``/ ! | ~ ^ % @ " ' ` ``, or a symbol outside of ASCII. So a
+  sigilless `s/2` is misread.
   `q`, `qq` and `Q` also take a bracket after blanks (`q {…}`), so a
   routine or a sigilless variable of your own named `q`, before a block
   or a subscript, is misread. The
@@ -202,12 +244,23 @@ Heuristics that can be wrong:
   as a method name, as a declared name, before `=>`, and for the
   keywords that are routines (`take`, `return`, `next`, `not`, …) right
   before a `(`: `take(1)`. A sigilless variable or a routine of your own
-  that is named like a keyword is highlighted as one.
+  that is named like a keyword is highlighted as one, and a declarator
+  that is called, as in `method($n)`, starts a declaration.
+- In a regex, a curly quote or a corner quote opens a quote, as Raku
+  has it. A lone one, as in `/‘/`, takes the text up to the next
+  closing quote on its line.
+- A `$/` in the replacement of `s/…/…/` is the match variable when a `/`
+  follows on the same line, and a `$` before the closing delimiter
+  otherwise.
+- A `:sym<…>` after the name of a declaration and blanks is part of the
+  name, also for a declaration that is not a regex.
 - A quote that does not interpolate (`'…'`, `q{…}`), a regex and a
   heredoc have to be closed. Otherwise their opener is read as ordinary
   code.
 - A quote that interpolates (`"…`, `qq[…`), a Pod block or an embedded
-  comment that is not closed runs to the end of the document.
+  comment that is not closed runs to the end of the document. A
+  declarator comment with brackets (`#|( …`) that is not closed ends
+  with its line.
 - The body of a regex declaration that is not closed runs up to the next
   line that starts a `token`, `rule` or `regex` declaration, or to the
   end of the document.
@@ -222,17 +275,18 @@ npm run dev  # serves demo/, an editor next to the syntax tree of its content
 
 The demo loads the sources, so a change to the grammar shows up on reload.
 
-To release a version:
-
-1. Run `npm version 1.2.3 --no-git-tag-version`, and turn the
-   `## Unreleased` heading of `CHANGELOG.md` into `## 1.2.3 (2026-01-31)`.
-   Commit and push.
-2. Run `npm publish` from that commit. It first runs the tests and checks
-   that `CHANGELOG.md` has that section and that nothing is uncommitted.
-3. Tag that commit `v1.2.3` and push the tag. A workflow then checks that
-   the tag matches `package.json` and that npm has the version, published
-   from that commit, and creates the GitHub release from the section of
-   `CHANGELOG.md`.
+Changes go in through pull requests. A pull request that changes what
+users get adds its entry under `## [Unreleased]` in `CHANGELOG.md`. One
+that adds a feature also adds a sample of it to
+`test/fixtures/new-in-<next minor version>.raku` (`new-in-0.5.raku`, say),
+and is labelled `enhancement` to make the next release a minor one.
+
+Releases are made with [tagpr](https://github.com/Songmu/tagpr). It keeps
+a release pull request open that bumps the version and turns
+`## [Unreleased]` into the version's section. Merging it tags the merge
+commit, and a workflow then publishes the version to npm with Trusted
+Publishing and creates the GitHub release from the section of
+`CHANGELOG.md`.
 
 ## License