rouge-raku 0.1.0 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 2abd02a3dd1ca15acdd031b967aa8b6276d8e10661664f9657806895788e7980
4
- data.tar.gz: 9e18210ce8e22ed1c47629aab3c92f0c6793a03c95cf52929268fb347efae27d
3
+ metadata.gz: b8ad4717eab72d4bd26053286b424815d08e0a8bcf2e3f7d1a486de53e9dbec3
4
+ data.tar.gz: 7b5af1c8f1ad38a46eca398e1d3c86533b1d6b4e35d47f7c3e4cd66735ea6d82
5
5
  SHA512:
6
- metadata.gz: 70bb9a66d622e2fde9f7eeabeba64c0133b1c2b25556a8442549f5f23781b586db555cd3be03fb3e52766864e5542e45cca997497643969d1dba8d2b1ee47549
7
- data.tar.gz: a92cf7b41df0f44c904af968299717d643224987fc11a2ccc1c8a2d53e297494b411fb0c43dc0b412089352c5104f629b7f394bf6e6e35d52dd38e1f2244b4f6
6
+ metadata.gz: 542d0cd2bc0c5b58bb51697571bbd2e61f92a4e02893e99e959d3ad0f3fb1a020bd264ffce1b760141ec75df2e843314c6a52456d77c3e0d4f3ba3e1a29c04b0
7
+ data.tar.gz: 2e32eafb117777f26128eee6dde13aa5e3e0146ab87f0b515876e3682b7cbe1db156eb975d4e0f84679e3995fd00369ee6deb2b6427f5ed2213a8be16a499b8b
data/CHANGELOG.md CHANGED
@@ -1,5 +1,44 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.2.0
4
+
5
+ - Quotes: `q.new` and `q'x` are names, `q(1)`, `m(1)` and `s(1)` are calls,
6
+ and `s[0]` is an index, none of them the start of a quote or a regex. A
7
+ quote interpolates what its adverbs say (`q:s`, `Qc`, `qq:!s`), `qww` and
8
+ `qqww` are quotes, and `:heredoc` opens a heredoc like `:to`.
9
+ - Strings interpolate calls and chains of method calls that end in
10
+ parentheses (`"$f()"`, `"$obj.name.uc()"`, `"@a>>.gist()"`) and subscripts
11
+ after a dot (`"@a.[0]"`).
12
+ - Regexes: recognised after `where`, `handles`, `...` and a flip-flop, and
13
+ across lines after a smartmatch. The arguments of a rule and of an adverb
14
+ are code (`<expr(3)>`, `m:nth(2)/x/`). `S!~~` and the other sequential
15
+ operators are not substitutions, and `[/]` is not a regex.
16
+ - Word lists: after a prefix operator (`~<a b c>`), after `enum`, as the
17
+ value of a colon-pair (`:name«$x»`), with an escaped `\>` in them, and
18
+ with `#` starting a comment in the interpolating forms.
19
+ - Numbers: `⅒`, `½` and `Ⅷ`, more forms in angle brackets (`<0x01/0x02>`,
20
+ `<NaN+0i>`), versions that end in a word (`v6.e.PREVIEW`), and no number
21
+ in `foo().5`.
22
+ - Names: extended identifiers are one variable (`$foo:bar<baz>`), `Z=>` is
23
+ the zip operator, a word after a minus sign keeps its meaning (`-Inf`),
24
+ `%` and `&` in front of a digit are operators (`$x%%3`), operator names
25
+ can be in double angle brackets, and the label after `next`, `last` and
26
+ `redo` is a label. A subscript in angle brackets is a string, not a part
27
+ of the variable (`%h<key>`).
28
+ - Hyper operators around a Unicode operator (`»⋅«`), and a hyper prefix in
29
+ front of a word list.
30
+ - Lists: the native types of NativeCall, the documented methods of the
31
+ metamodel, three more traits, the phasers that can follow `will`, and the
32
+ values of the enumerations that come with the language (`Less`, `Empty`,
33
+ `SIGINT`, ...).
34
+ - The `#!` line that starts a script is a hashbang comment.
35
+ - Pod: code blocks are lexed as Raku unless they name another language
36
+ (`:lang<shell>`), and the other verbatim blocks are plain text.
37
+ Formatting codes can use `<< >>` and `« »`, nest and span lines. The
38
+ options of a directive are lexed as pairs, and the name of a block is
39
+ part of its directive (`=begin comment`). Pod can be indented, and can
40
+ end with the file.
41
+
3
42
  ## 0.1.0
4
43
 
5
44
  First release.
@@ -42,10 +42,15 @@ module Rouge
42
42
 
43
43
  # names of the traits that can follow "is"
44
44
  TRAITS = %w(
45
- DEPRECATED assoc built cached copy default dynamic equiv
46
- export hidden-from-USAGE hidden-from-backtrace
47
- implementation-detail looser native nodal pure raw readonly
48
- repr required rw symbol test-assertion tighter
45
+ assoc built cached copy default DEPRECATED dynamic encoded
46
+ equiv export hidden hidden-from-backtrace hidden-from-USAGE
47
+ implementation-detail looser native nativesize nodal pure raw
48
+ readonly repr required rw symbol test-assertion tighter
49
+ ).freeze
50
+
51
+ # names of the phasers that can follow "will"
52
+ WILL_TRAITS = %w(
53
+ begin check end enter first keep leave undo
49
54
  ).freeze
50
55
 
51
56
  # Routines that can be called as a plain word: say "hi", map(...). These
@@ -88,13 +93,14 @@ module Rouge
88
93
  # "$x.name", but on its own it is a word the user chose ("name => 1",
89
94
  # "sub name"), so these are only builtins after a dot.
90
95
  METHODS = %w(
91
- abs2rel absolute accept ACCEPTS accessed acquire act action
92
- actions add add_attribute add_enum_value add_fallback
93
- add_method add_parent add_private_method add_role add_trustee
94
- adverb after allocate allof alternative-names annotations
95
- antipair antipairs anyof app_lifetime arch archname are args
96
- arity ASSIGN-KEY ASSIGN-POS assuming ast at AT-KEY AT-POS
97
- attributes auth backtrace base base-repeating basename batch
96
+ abs2rel absolute accept ACCEPTS accepts_type accessed acquire
97
+ act action actions add add_attribute add_enum_value
98
+ add_fallback add_method add_parent add_private_method add_role
99
+ add_stash add_trustee adverb after allocate allof
100
+ alternative-names annotations antipair antipairs anyof api
101
+ app_lifetime arch archetypes archname are args arity
102
+ ASSIGN-KEY ASSIGN-POS assuming ast at AT-KEY AT-POS attributes
103
+ auth backtrace base base-repeating base_type basename batch
98
104
  before BIND-KEY BIND-POS bind-stderr bind-stdin bind-stdout
99
105
  bind-udp bits bless block bool-only bounds break Bridge broken
100
106
  BUILD build-date bytecode-size bytes cache CALL-ME
@@ -102,57 +108,67 @@ module Rouge
102
108
  caption capture catdir categorize-list catfile catpath cause
103
109
  changed child child-name child-typename chunks classify-list
104
110
  cleanup clone close-stdin closed code codes collate column
105
- command comment compiler compose compose_type composer concise
106
- condition config configure_destroy configure_type_checking
107
- conj connect constraints construct contains contents count
108
- count-only cpu-cores cpu-usage CREATE create_type created cue
109
- curdir curupdir d day day-fraction day-of-month day-of-week
111
+ command comment compiler composalizer compose compose_type
112
+ compose_values composer compute_mro concise condition config
113
+ configure_destroy configure_type_checking conj connect
114
+ constraints construct contains contents count count-only
115
+ cpu-cores cpu-usage CREATE create_type created cue curdir
116
+ curupdir d day day-fraction day-of-month day-of-week
110
117
  day-of-year daycount days-in-month days-in-year dd-mm-yyyy
111
- declaration decode decoder default DEFINITE delayed DELETE-KEY
118
+ declaration decode decoder default DEFINITE definite delayed
119
+ delegate_methods_to delegating_methods_to DELETE-KEY
112
120
  DELETE-POS denominator desc DESTROY destroyers dev devnull
113
121
  devtype did-you-mean dir-sep dir-with-entries dirname
114
122
  DISTROnames do does dynamic e eager earlier enclosing encode
115
123
  encoder encoding ends-with enum_from_value enum_value_list
116
124
  enum_values enums eof err exception excludes-max excludes-min
117
- EXISTS-KEY EXISTS-POS exitcode expected extension f feature
118
- file filename find_method find_method_qualified finish
119
- first-date-in-month flatmap flush fmt format formatter freeze
120
- from from-list from-loop from-posix from-slurpy full get_value
121
- got grab grabpairs handle handled handles hardware
122
- has_accessor headers hh-mm-ss hidden hides hour HOW how hyper
123
- id illegal im in in-range in-timezone indent indices infinite
124
- infix inode install_method_cache instead int-bounds interval
125
- invalid-str invert invocant is-absolute is-hidden
126
- is-implementation-detail is-initial-thread is-int is-lazy
127
- is-leap-year is-relative is-routine is-setting is-win
128
- is_trusted is_type isa isNaN iterator julian-date keep kept
129
- KERNELnames key keyof kill kxxv l lang last last-date-in-month
130
- later lazy leading level line listen live local lock lookup
131
- made make match maxpairs merge message method method_table
132
- methods migrate minpairs minus minute misplaced mm-dd-yyyy
133
- mode modified modified-julian-date modifier month mro multi
134
- multi-invocant multiness my name named named_names narrow
135
- native-descriptor new new-from-daycount new-from-pairs
136
- new_type next next-handle next-interesting-index nice nl-in
137
- nl-out norm nude numerator of offset offset-in-hours
138
- offset-in-minutes old on-close on-switch opened operation
139
- optional orig os-error osname out out-buffer outer
140
- outer-caller-idx package package-kind package-name packages
141
- pair pairup parameter params parent parent-name parents parse
125
+ EXISTS-KEY EXISTS-POS exitcode expected export_callback
126
+ extension f feature file filename find_method
127
+ find_method_qualified find_private_method finish
128
+ first-date-in-month flatmap flush flush_cache fmt format
129
+ formatter freeze from from-list from-loop from-posix
130
+ from-slurpy full generate_mixin get_value got grab grabpairs
131
+ handle handled handles hardware has_accessor headers hh-mm-ss
132
+ hidden hides hour HOW how hyper id illegal im in in-range
133
+ in-timezone indent indices infinite infix inode
134
+ install_method_cache instead int-bounds interval invalid-str
135
+ invert invocant is-absolute is-hidden is-implementation-detail
136
+ is-initial-thread is-int is-lazy is-leap-year is-relative
137
+ is-routine is-setting is-win is_composed is_mixin is_trusted
138
+ is_type isa isNaN iterator julian-date keep kept KERNELnames
139
+ key keyof kill kxxv l lang last last-date-in-month later lazy
140
+ leading level line listen live local lock lookup made make
141
+ match maxpairs merge message method method_table methods
142
+ migrate minpairs minus minute misplaced mixin mixin_attribute
143
+ mm-dd-yyyy mode modified modified-julian-date modifier month
144
+ mro mro_unhidden multi multi-invocant multiness my name named
145
+ named_names narrow native-descriptor new new-from-daycount
146
+ new-from-pairs new_type next next-handle
147
+ next-interesting-index nice nl-in nl-out nominalize norm nude
148
+ numerator of offset offset-in-hours offset-in-minutes old
149
+ on-close on-switch opened operation optional orig os-error
150
+ osname out out-buffer outer outer-caller-idx package
151
+ package-kind package-name packages pair pairup parameter
152
+ parameterize_type params parent parent-name parents parse
142
153
  parsefile parts path path-sep payload peer-host peer-port perl
143
154
  phaser pickpairs pid placeholder plus polar poll polymod port
144
155
  pos positional posix postfix postmatch precomp-ext
145
- precomp-target pred prefix prematch print-nl print-to private
146
- private_method_table proc protect pull-one push-all
147
- push-at-least push-exactly push-until-lazy qualifier-type quit
148
- r race radix raku range raw re read readchars readonly ready
149
- reallocate reals reason rebless receive recv redispatcher redo
150
- rel2abs relative release remove replace-with replacement REPR
151
- reserved resolve restore result resume rethrow returns right
152
- role roles_to_compose rolish rootdir rotor routine-type rw rwx
153
- s schedule-on scheduler scope second seek send serial
154
- set-instruments set_hidden set_name set_package set_rw
155
- set_value setup_finalization shape share sibling sigil signals
156
+ precomp-target pred prefix prematch pretend_to_be
157
+ pretending_to_be print-nl print-to private
158
+ private_method_names private_method_table private_methods proc
159
+ protect publish_method_cache pull-one push-all push-at-least
160
+ push-exactly push-until-lazy qualifier-type quit r race radix
161
+ raku range raw re read readchars readonly ready reallocate
162
+ reals reason rebless receive recv redispatcher redo rel2abs
163
+ relative release remove replace-with replacement REPR reserved
164
+ resolve restore result resume rethrow returns right role
165
+ roles_to_compose rolish rootdir rotor routine-type rw rwx s
166
+ schedule-on scheduler scope second seek send serial
167
+ set-instruments set_api set_auth set_composalizer
168
+ set_export_callback set_hidden set_is_mixin
169
+ set_mixin_attribute set_name set_package set_parameterizer
170
+ set_rw set_value set_ver set_why setup_finalization
171
+ setup_mixin_cache shape share shortname sibling sigil signals
156
172
  signature sink sink-all skip-at-least skip-at-least-pull-one
157
173
  skip-one slice slurp-rest slurpy socket-host socket-port
158
174
  source source-package spawn SPEC splitdir splitpath stable
@@ -161,8 +177,9 @@ module Rouge
161
177
  suffix summary t tap target target-name tell then throttle
162
178
  throw timezone tmpdir to to-posix today toggle total trailing
163
179
  trans tree truncated-to trusts try_acquire trying twigil type
164
- type_captures typename udp uncaught_handler univals unlock
165
- unset unwrap updir usage-name utc value VAR variable
180
+ type_captures type_check type_parameter_at type_parameterized
181
+ type_parameters typename udp uncaught_handler univals unlock
182
+ unset unwrap updir usage-name utc value VAR variable ver
166
183
  verbose-config version VMnames volume vow w wait watch
167
184
  watch-path week week-number week-year weekday-of-month WHAT
168
185
  when WHERE WHEREFORE WHICH WHO whole-second WHY workaround
@@ -224,7 +241,7 @@ module Rouge
224
241
  Routine::WrapHandle Sequence Signal
225
242
  Telemetry::Instrument::ThreadPool Unicode ValueObjAt array
226
243
  blob8 blob16 blob32 blob64 buf8 buf16 buf32 buf64 byte int num
227
- str uint
244
+ str uint bool ssize_t ulong ulonglong utf16 void
228
245
  ).freeze
229
246
 
230
247
  WORD_OPERATORS = %w(
@@ -233,8 +250,15 @@ module Rouge
233
250
  orelse R so unicmp x X xor xx Z
234
251
  ).freeze
235
252
 
253
+ # terms that stand for a value, and the values of the enumerations
254
+ # that come with the language
236
255
  CONSTANTS = %w(
237
- π τ ∞ 𝑒 pi tau Inf NaN e i
256
+ π τ ∞ 𝑒 pi tau Inf NaN e i BigEndian Broken Empty FileChanged
257
+ FileRenamed IterationEnd Kept Less LittleEndian More
258
+ NativeEndian PF_INET PF_INET6 Planned Same SeekFromBeginning
259
+ SeekFromCurrent SeekFromEnd SIGABRT SIGALRM SIGBREAK SIGBUS
260
+ SIGHUP SIGINT SIGKILL SIGTERM SIGUSR1 SOCK_DGRAM SOCK_PACKET
261
+ SOCK_RAW SOCK_RDM SOCK_SEQPACKET SOCK_STREAM
238
262
  ).freeze
239
263
 
240
264
  # symbolic operators, including the Unicode synonyms
@@ -267,18 +291,30 @@ module Rouge
267
291
  {} ⦅⦆ 「」
268
292
  ).to_h(&:chars).freeze
269
293
 
294
+ # what the text of a Pod formatting code looks like
295
+ POD_FORMATS = {
296
+ 'B' => Generic::Strong, 'I' => Generic::Emph, 'C' => Str::Backtick,
297
+ 'K' => Str::Backtick, 'T' => Str::Backtick, 'V' => Str::Backtick,
298
+ }.freeze
299
+
270
300
  # --- building blocks for the rules below
271
301
 
302
+ # the lines of a Pod paragraph: up to a blank line or a directive
303
+ pod_paragraph = '(?:(?![ \t]*$|[ \t]*=[A-Za-z])[^\n]*\n?)*'
304
+
272
305
  # Ruby's \w, \d and \s are ASCII-only. Raku is not, so the rules use
273
306
  # \p{Nd} and \p{Space}, and word characters are spelled out.
274
- w = '\p{L}\p{N}_'
307
+ w = W = '\p{L}\p{N}_'
275
308
  # Superscript digits and signs are exponents ("$x²"), so they must
276
309
  # not be part of an identifier.
277
310
  sup = '²³¹⁰-⁻'
278
311
  word = "[#{w}&&[^#{sup}]]"
279
312
  nondigit = "[\\p{L}\\p{No}\\p{Nl}_&&[^#{sup}]]"
280
- # one identifier character, for lookbehinds
281
- ident_char = "[#{w}'\\-:&&[^#{sup}]]"
313
+ # Not in the middle of a longer word: after a word character or a
314
+ # colon, or after the hyphen or apostrophe that joins two parts of
315
+ # an identifier (my-if, don't). A minus sign on its own is not one:
316
+ # -Inf
317
+ not_in_word = "(?<![#{w}:&&[^#{sup}]]|[#{w}&&[^#{sup}]]['\\-])"
282
318
  # What may follow a word for it to still be part of a longer
283
319
  # identifier. A single colon does not continue an identifier
284
320
  # ("@a.push: 1", "Blob:D:"), and neither does a hyphen that is not
@@ -290,22 +326,37 @@ module Rouge
290
326
  # an identifier; a hyphen or apostrophe must be followed by a letter
291
327
  ident = "#{nondigit}#{word}*(?:['\\-]#{nondigit}#{word}*)*"
292
328
  qualified_ident = "#{ident}(?:::#{ident})*"
329
+ # A number can start with its decimal point (.5), but not right after
330
+ # a term: in "foo().5" and "@a[0].5" the dot is not part of a number.
331
+ # (it looks ahead for the dot first, so as not to look behind at
332
+ # every character)
333
+ leading_dot = "(?=\\.)(?<![#{w})\\]}])\\."
334
+ # one part of a complex number in angle brackets: <1.5e0+Inf\i>
335
+ angle_number = '(?:\p{Nd}+(?:\.\p{Nd}+)?(?:e[-+]?\p{Nd}+)?|\.\p{Nd}+|NaN|Inf)'
293
336
  # the operator part of a routine name: infix:<+>, circumfix:«[ ]»
294
- op_name_suffix = '(?::(?:sym)?(?:<[^>\n]+>|«[^»\n]+»|' \
337
+ op_name_suffix = '(?::(?:sym)?(?:<<(?!=>)(?:(?!>>)[^\n;])+>>|<[^>\n]+>|«[^»\n]+»|' \
295
338
  '\[(?:"[^"\n]*"|\'[^\'\n]*\'|[^\]\n])+\]))'
296
339
  routine_name = "[!^]?#{qualified_ident}#{op_name_suffix}?"
297
340
  op_categories = '(?:infix|prefix|postfix|circumfix|postcircumfix|term|trait_mod)'
298
341
  type_name = "[A-Z]#{word}*(?:['\\-]#{nondigit}#{word}*)*(?:::#{ident})*"
299
342
  # type smileys: Int:D, Foo:U, Any:_
300
343
  smiley = "(?::[UD_](?![#{w}'\\-]))"
301
- after_dot = '(?:(?<=\.)(?<!\.\.)|(?<=\.[\^?&+*]))'
302
- subscript = '(?:\[[^\]\n]*\]|\{[^}\n]*\}|<[^>\n]*>)'
344
+ after_dot = '(?:(?<=\.)(?<!\.\.)|(?<=\.[\^?&+*])(?<!\.\.[\^?&+*]))'
345
+ subscript = '(?:(?:>>|»)?\.)?(?:\[[^\]\n]*\]|\{[^}\n]*\}|<[^>\n]*>|«[^»\n]*»)'
303
346
  # the arguments of a call, with one level of nested parentheses;
304
347
  # a parenthesis in a quoted string does not count
305
348
  # (atomic and possessive, so that it can not backtrack for ages)
306
349
  arguments = '\((?>\'[^\'\n]*\'|"[^"\n]*"|[^()\n]|\([^()\n]*\))*+\)'
307
350
  # what can follow a variable in a string: a subscript or a method call
308
- postfix = "(?:#{subscript}|\\.[\\^?&]?#{ident}#{arguments})"
351
+ # A chain of method calls counts as long as it ends in parentheses
352
+ # ("$x.name.uc()", "@a>>.gist()"), and so does a call of the
353
+ # variable ("$f()").
354
+ postfix = "(?:#{subscript}|#{arguments}|(?:(?:>>|»)?\\.[\\^?&]?#{ident})+#{arguments})"
355
+ # the rest of an extended identifier: $foo:bar<baz>, $take-me:['home'],
356
+ # $a:b<c>:d:e<f>. Parentheses only belong to it after a name:
357
+ # &code:(Int) has a signature.
358
+ extended = "(?:(?::#{ident})*:(?:#{ident})?(?:<[^>\\n]*>|«[^»\\n]*»|\\[[^\\]\\n]*\\]|" \
359
+ "(?<=[#{w}])\\([^)\\n]*\\)))*"
309
360
  # %h<key>, %h<<$key>>, %h«key»; kept to one line so that a stray '<'
310
361
  # in code can't swallow what follows it
311
362
  angle_subscripts = '(?:<<[^>\n]*>>|<[^>\n]*>|«[^»\n]*»)*'
@@ -315,9 +366,9 @@ module Rouge
315
366
  regex_words = %w(
316
367
  say put print when if elsif unless while until given with without
317
368
  and or not so return grep map first split comb match subst
318
- contains ff fff xor andthen orelse
369
+ contains ff fff xor andthen orelse where handles
319
370
  )
320
- regex_position = "(?=\\p{Space}*\\/)(?:(?<=[=(,{\\[;:!~|&?])|(?<==>)|" \
371
+ regex_position = "(?=\\p{Space}*\\/)(?:(?<=[=(,{\\[;:!~|&?^…])|(?<==>)|(?<=\\.\\.\\.)|" \
321
372
  "#{regex_words.map { |x| "(?<=\\b#{x} )" }.join('|')})"
322
373
  open_brackets = Regexp.escape(BRACKETS.keys.join)
323
374
 
@@ -326,12 +377,22 @@ module Rouge
326
377
  quote_start = "(?<![#{w}'\\-.])"
327
378
  # the adverbs of a quote: q:to, qq:!c, m:i:g
328
379
  adverbs = "(?:\\p{Space}*:!?[#{w}]+)+"
380
+ # ... and of a regex, which can have arguments: m:nth(2)
381
+ regex_adverbs = "(?:\\p{Space}*:!?[#{w}]+(?:\\([^)\\n]*\\))?)+"
382
+ # What can delimit a quote, besides a bracket. Most punctuation
383
+ # could, but in "q.new" and "q = 1" q is a name. An apostrophe in
384
+ # front of a letter is part of an identifier (q'x), unless adverbs
385
+ # came first (q:to 'END'), which the rule captures as its group
386
+ # number +adverbs+.
387
+ quote_delimiter = lambda do |adverbs|
388
+ "(?:[\\/!|\"~^%@`§]|(?(#{adverbs})'|'(?![\\p{L}_])))"
389
+ end
329
390
  # Between a quote word and its delimiter, whitespace can only come
330
391
  # before a bracket or a slash: in "q ~~ $x" and "m ?? 1 !! 2", q and
331
392
  # m are names.
332
393
  before_delimiter = "(?:\\p{Space}*(?=[#{open_brackets}\\/])|)"
333
394
  # where a "<" starts a word list rather than being a comparison
334
- term_position = "(?=\\p{Space}*<)(?:(?<=[=(,\\[])|(?<=\\bfor ))"
395
+ term_position = "(?=\\p{Space}*<)(?:(?<=[=(,\\[])|(?<=\\bfor )|(?<=\\benum )|(?<=:: ))"
335
396
  # "token", "rule" and "regex" are ordinary words in "$x.rule",
336
397
  # "rule => 1" or "/regex/"
337
398
  not_a_regex_declarator = "(?<![#{w}'.:$@%&\\/<\\-])"
@@ -357,11 +418,11 @@ module Rouge
357
418
  end
358
419
 
359
420
  word_match = lambda do |words, suffix = ''|
360
- "(?<!#{ident_char})(#{any_of.(words)})" \
421
+ "#{not_in_word}(#{any_of.(words)})" \
361
422
  "#{suffix}(?!#{ident_end})"
362
423
  end
363
424
 
364
- not_a_builtin_type = "(?!(?:#{any_of.(BUILTIN_CLASSES)}|True|False|Nil)" \
425
+ not_a_builtin_type = "(?!(?:#{any_of.(BUILTIN_CLASSES)}|True|False|Nil|Inf|NaN)" \
365
426
  "#{smiley}?(?!#{ident_end}))"
366
427
  # After a dot every builtin routine is a builtin, and so is the name
367
428
  # of a type: "$x.Int" calls the coercion method, it does not name
@@ -370,10 +431,23 @@ module Rouge
370
431
  builtins = Set.new(BUILTINS)
371
432
 
372
433
  OPEN_BRACKET = /[#{open_brackets}]/
373
- HEREDOC_OPENER = /#{nw}(qq|q|Q)[a-zA-Z]?\p{Space}*((?::[#{w}]+\p{Space}*)+)([^#{w}\p{Space}:])/
434
+ HEREDOC_OPENER = /#{nw}(qq|q|Q)(ww|[a-zA-Z])?\p{Space}*((?::!?[#{w}]+\p{Space}*)+)([^#{w}\p{Space}:])/
435
+ HEREDOC_ADVERB = /:(?:to|heredoc)\b/
436
+ # What a quote can interpolate: scalars, arrays, hashes, function
437
+ # calls, closures and backslash escapes. Each has an adverb that
438
+ # turns it on, with a short and a long name: q:s, q:scalar
439
+ INTERPOLATIONS = {
440
+ 's' => :s, 'scalar' => :s, 'a' => :a, 'array' => :a, 'h' => :h, 'hash' => :h,
441
+ 'f' => :f, 'function' => :f, 'c' => :c, 'closure' => :c,
442
+ 'b' => :b, 'backslash' => :b,
443
+ }.freeze
444
+ EVERY_INTERPOLATION = INTERPOLATIONS.values.uniq.freeze
445
+ SIGIL_INTERPOLATIONS = { '$' => :s, '@' => :a, '%' => :h, '&' => :f }.freeze
374
446
 
375
447
  start do
376
448
  @brace_levels = []
449
+ @interpolations = EVERY_INTERPOLATION
450
+ @pod_formats = []
377
451
  end
378
452
 
379
453
  # Consumes the rest of a construct opened with the delimiter +opener+
@@ -458,18 +532,32 @@ module Rouge
458
532
  token Str::Regex, closing
459
533
  end
460
534
 
535
+ # Lexes what opens a regex: its +word+ (m, rx, s), its adverbs, if
536
+ # it has any, and its +opener+. Adverbs that have arguments are
537
+ # lexed as code, for the sake of the arguments: m:nth(2)/x/
538
+ def lex_regex_opening(opening, word, opener)
539
+ adverbs = opening[word.length...opening.length - opener.length]
540
+ return token(Str::Regex, opening) unless adverbs.include?('(')
541
+
542
+ token Str::Regex, word
543
+ sublex adverbs
544
+ # (a copy, as Rouge appends the tokens that follow to this string,
545
+ # and the opener is still needed to find where the regex ends)
546
+ token Str::Regex, opener.dup
547
+ end
548
+
461
549
  # s/pattern/replacement/, s{pattern}{replacement}, tr/from/to/, ...
462
550
  def lex_substitution(stream, opening, kind, opener)
463
551
  transliteration = kind.downcase == 'tr'
464
552
 
465
- token Str::Regex, opening
553
+ lex_regex_opening(opening, kind, opener)
466
554
  lex_regex_part(stream, opener, transliteration ? nil : :regex_body)
467
555
 
468
556
  replacement = transliteration ? nil : :interpolated
469
557
  if BRACKETS[opener[0]].nil?
470
558
  # s/a/b/ : the replacement follows directly, with the same delimiter
471
559
  lex_regex_part(stream, opener, replacement, match_variable: true)
472
- elsif stream.check(/\p{Space}*#{OPEN_BRACKET}/)
560
+ elsif stream.check(/\p{Space}*#{OPEN_BRACKET}/o)
473
561
  # s{a}{b} : another bracketed group, possibly after whitespace.
474
562
  # Otherwise it is s{a} = b, an ordinary assignment.
475
563
  token Text::Whitespace, stream.scan(/\p{Space}*/)
@@ -479,17 +567,103 @@ module Rouge
479
567
  end
480
568
  end
481
569
 
570
+ # Whether what follows the opening "s[" is a substitution, s[a] = 'b',
571
+ # s[a] += 1 or s[a][b], rather than an index into something called
572
+ # s: s[0]
573
+ def substitution_follows?(stream, opener)
574
+ start = stream.pos
575
+ _, closing = scan_regex(stream, opener)
576
+ found = !closing.empty? && stream.match?(/\p{Space}*(?:[-+*\/%~|&^]{0,2}=(?![=>~:])|#{OPEN_BRACKET})/o)
577
+ stream.pos = start
578
+ found
579
+ end
580
+
581
+ # Lexes a Pod block whose content is not Pod: its first line, of
582
+ # which +opening+ are the tokens up to the +name+ of the block and
583
+ # +config+ is what follows it, and then its +body+. Code is Raku,
584
+ # unless the block says that it is something else: =begin code
585
+ # :lang<shell>
586
+ def lex_pod_verbatim(name, config, body, opening)
587
+ token Text::Whitespace, opening[0]
588
+ token Comment::Preproc, opening[1]
589
+ token Text::Whitespace, opening[2]
590
+ token Comment::Preproc, name
591
+ sublex config, :pod_config_line
592
+
593
+ if name == 'code' && !config.match?(/:lang<(?!raku>)/)
594
+ sublex body
595
+ else
596
+ token Comment::Multiline, body
597
+ end
598
+ end
599
+
600
+ # Opens the Pod formatting code +letter+, as in B<bold>. What closes
601
+ # it, how its text looks and how many angle brackets are open inside
602
+ # it are kept until it is closed.
603
+ def open_pod_format(letter, opener)
604
+ token Name::Decorator, letter
605
+ token Punctuation, opener
606
+ closer = BRACKETS[opener[0]] * opener.length
607
+ @pod_formats << [closer, POD_FORMATS.fetch(letter, Comment::Multiline), 0]
608
+ push :pod_format
609
+ end
610
+
611
+ # Lexes a variable and the subscripts in angle brackets that follow
612
+ # it, which are quoted words, not a part of its name: %h<key>,
613
+ # %h<<$key>>, %h«$key»
614
+ def lex_variable(type, name, subscripts)
615
+ token type, name
616
+ subscripts.scan(/<<[^>\n]*>>|«[^»\n]*»|<[^>\n]*>/) do |subscript|
617
+ if subscript.start_with?('<<', '«')
618
+ sublex subscript
619
+ else
620
+ token Str, subscript
621
+ end
622
+ end
623
+ end
624
+
482
625
  # Lexes +text+ on its own, starting in +state+.
483
626
  def sublex(text, state = :root)
484
627
  lexer = self.class.new(options)
485
628
  lexer.reset!
486
629
  lexer.push(state) unless state == :root
630
+ yield lexer if block_given?
487
631
  delegate(lexer, text)
488
632
  end
489
633
 
634
+ # What the quote +word+ interpolates, given the letter that follows
635
+ # it (Qs) and its +adverbs+ (q:s, qq:!c).
636
+ def interpolations_of(word, letter, adverbs)
637
+ found = word == 'qq' ? EVERY_INTERPOLATION.dup : []
638
+ found << INTERPOLATIONS[letter] if INTERPOLATIONS.key?(letter)
639
+ adverbs.scan(/:(!?)([#{W}]+)/o) do |negated, name|
640
+ kinds = %w(qq double).include?(name) ? EVERY_INTERPOLATION : [INTERPOLATIONS[name]].compact
641
+ found = negated.empty? ? found | kinds : found - kinds
642
+ end
643
+ found
644
+ end
645
+
646
+ # Lexes the +body+ of a quote, in which only +interpolations+ are
647
+ # special.
648
+ def lex_quoted(body, interpolations)
649
+ if interpolations.empty?
650
+ token Str, body
651
+ else
652
+ sublex(body, :interpolated) { |lexer| lexer.interpolations = interpolations }
653
+ end
654
+ end
655
+
656
+ attr_writer :interpolations
657
+
658
+ # Whether the string being lexed interpolates +kind+. Code inside a
659
+ # string is code like any other, and so are the strings in it.
660
+ def interpolates?(kind)
661
+ !@brace_levels.empty? || @interpolations.include?(kind)
662
+ end
663
+
490
664
  # Lexes what follows the opening quote of a heredoc: the rest of
491
665
  # that line, which is ordinary code, and then the heredoc's body.
492
- def lex_heredocs(stream, terminator, interpolate)
666
+ def lex_heredocs(stream, terminator, interpolations)
493
667
  rest_of_line = stream.scan(/[^\n]*/)
494
668
  sublex rest_of_line
495
669
  return unless stream.scan(/\n/)
@@ -497,8 +671,8 @@ module Rouge
497
671
  token Text::Whitespace, "\n"
498
672
 
499
673
  # the bodies follow in the order their heredocs were opened
500
- heredocs = [[terminator, interpolate]] + extra_heredocs(rest_of_line)
501
- heredocs.each_with_index do |(name, interpolates), index|
674
+ heredocs = [[terminator, interpolations]] + extra_heredocs(rest_of_line)
675
+ heredocs.each_with_index do |(name, interpolated), index|
502
676
  last_line = /^[ \t]*#{Regexp.escape(name)}[ \t]*$/
503
677
  body = stream.scan_until(/(?=#{last_line})/)
504
678
  if body.nil?
@@ -506,11 +680,7 @@ module Rouge
506
680
  stream.terminate
507
681
  end
508
682
 
509
- if interpolates
510
- sublex body, :interpolated
511
- else
512
- token Str, body
513
- end
683
+ lex_quoted(body, interpolated)
514
684
  break if stream.eos?
515
685
 
516
686
  token Str, stream.scan(last_line)
@@ -521,23 +691,27 @@ module Rouge
521
691
  end
522
692
 
523
693
  # Heredocs opened later on the same line, as pairs of terminator
524
- # and whether the body interpolates.
694
+ # and what the body interpolates.
525
695
  def extra_heredocs(line)
526
696
  found = []
527
697
  pos = 0
528
698
  while (match = HEREDOC_OPENER.match(line, pos))
529
699
  pos = match.end(0)
530
- quote, adverbs, opener = match.captures
531
- next unless adverbs.match?(/:to\b/)
700
+ quote, letter, adverbs, opener = match.captures
701
+ next unless adverbs.match?(HEREDOC_ADVERB)
532
702
 
533
703
  stop = line.index(BRACKETS.fetch(opener, opener), pos)
534
704
  next unless stop
535
705
 
536
- found << [line[pos...stop], quote == 'qq' || adverbs.include?(':qq')]
706
+ found << [line[pos...stop], interpolations_of(quote, letter, adverbs)]
537
707
  end
538
708
  found
539
709
  end
540
710
 
711
+ # The rules are built the first time their state is used, and their
712
+ # blocks can see the patterns above. A block that assigned to one of
713
+ # those names would change the rules that are built after it runs.
714
+ #
541
715
  # If you're modifying these rules, be careful if you need to process
542
716
  # '{' or '}' characters. Code can be nested in strings, and the
543
717
  # :embedded state counts braces to know where that code ends, so if
@@ -547,12 +721,16 @@ module Rouge
547
721
  # rules after them skip.
548
722
  # Pod: everything after =finish, a delimited block (=begin pod ...
549
723
  # =end pod), or a paragraph or abbreviated block, which ends at the
550
- # first blank line
551
- rule %r/^=finish\b.*|^(\p{Space}*)=begin\p{Space}+([#{w}]+)\b.*?^\1=end\p{Space}+\2|^\p{Space}*=for.*?\n\p{Space}*?\n|^=.*?\n\p{Space}*?\n/m do |m|
724
+ # first blank line or with the file. A delimited block can be
725
+ # indented, and so can a paragraph; an abbreviated one can not, as
726
+ # a line of code may well start with "=".
727
+ rule %r/^=finish\b.*|^([ \t]*)=begin\p{Space}+([#{w}]+)\b.*?^\1=end\p{Space}+\2|^[ \t]*=for\b.*?(?:\n\p{Space}*?\n|\z)|^=[A-Za-z].*?(?:\n\p{Space}*?\n|\z)/m do |m|
552
728
  sublex m[0], :pod_body
553
729
  end
554
730
  # a regex without m or rx: $s ~~ /x/, .subst(/x/, ''), say /x/
555
- rule %r/#{regex_position}(\p{Space}*)(\/)(?!\/)(?=(?:\\[\s\S]|[^\/\\\n])*\/)/ do |m|
731
+ # It has to end on the line it starts on, as the "/" may be a
732
+ # division after all, except after a smartmatch, where it can not.
733
+ rule %r/#{regex_position}(\p{Space}*)(\/)(?![\/\]])(?:(?<=~~\/|~~\p{Space}\/)|(?=(?:\\[\s\S]|[^\/\\\n])*\/))/ do |m|
556
734
  opener = m[2]
557
735
  groups Text::Whitespace, Str::Regex
558
736
  lex_regex_part(m, opener, :regex_body)
@@ -568,7 +746,9 @@ module Rouge
568
746
  # --- whitespace and punctuation. No rule below starts with either,
569
747
  # and they are the most common tokens, so they are not made to
570
748
  # wait for all the other rules to fail first.
571
- rule %r/\p{Space}+/, Text::Whitespace
749
+ # (whitespace stops at the start of a line that opens a Pod block,
750
+ # for the rule above to see the block from there)
751
+ rule %r/\p{Space}*\n(?=[ \t]+=(?:begin|for)\p{Space})|\p{Space}+/, Text::Whitespace
572
752
  rule %r/[;,()\]]/, Punctuation
573
753
 
574
754
  # --- comments
@@ -577,22 +757,34 @@ module Rouge
577
757
  token Comment::Multiline, opening + scan_delimited(m, m[1]).join
578
758
  end
579
759
  rule %r/#[|=].*/, Comment::Special
760
+ # the first line of a script: #!/usr/bin/env raku
761
+ rule %r/\A#!.*/, Comment::Hashbang
580
762
  rule %r/#.*/, Comment::Single
581
763
 
582
764
  # --- variables. Nothing else starts with a sigil either.
583
765
  # attributes ($!x, $.x) and compile-time / pod variables ($?FILE, $=pod)
584
- rule %r/[$@%&][.!]#{qualified_ident}#{angle_subscripts}/, Name::Variable::Instance
585
- rule %r/[$@%&][?=]#{ident}#{angle_subscripts}/, Name::Variable::Magic
766
+ rule %r/([$@%&][.!]#{qualified_ident})(#{angle_subscripts})/ do |m|
767
+ lex_variable(Name::Variable::Instance, m[1], m[2])
768
+ end
769
+ rule %r/([$@%&][?=]#{ident})(#{angle_subscripts})/ do |m|
770
+ lex_variable(Name::Variable::Magic, m[1], m[2])
771
+ end
586
772
  rule %r/::\?[#{w}]+/, Name::Variable::Global
587
- rule %r/[$@%&]\*#{qualified_ident}#{angle_subscripts}/, Name::Variable::Global
588
- rule %r/\$[!\/¢]#{angle_subscripts}/, Name::Variable::Global
773
+ rule %r/([$@%&]\*#{qualified_ident})(#{angle_subscripts})/ do |m|
774
+ lex_variable(Name::Variable::Global, m[1], m[2])
775
+ end
776
+ rule %r/(\$[!\/¢])(#{angle_subscripts})/ do |m|
777
+ lex_variable(Name::Variable::Global, m[1], m[2])
778
+ end
589
779
  # the & sigil names a routine: &infix:<+>, &squared, &squared(2),
590
780
  # which may be one that comes with the language: &elems
591
781
  rule %r/&#{op_categories}#{op_name_suffix}/, Name::Function
592
782
  rule %r/&(?:::)?(#{qualified_ident})/ do |m|
593
783
  token builtins.include?(m[1]) ? Name::Builtin : Name::Function
594
784
  end
595
- rule %r/[$@%&][\^:~]?(?:::)?(?:#{qualified_ident}|\p{Nd}+)#{angle_subscripts}/, Name::Variable
785
+ rule %r/([$@%&][\^:~]?(?:::)?#{qualified_ident}#{extended}|[$@]\p{Nd}+)(#{angle_subscripts})/ do |m|
786
+ lex_variable(Name::Variable, m[1], m[2])
787
+ end
596
788
  rule %r/\$(?:<[^>\n]*>)+/, Name::Variable
597
789
  # anonymous variables: "state $ = 0", "$++"
598
790
  rule %r/[$@](?=[\p{Space}=;,)\]]|\+\+|--)/, Name::Variable
@@ -614,13 +806,15 @@ module Rouge
614
806
  # radix literals: :16<FF>
615
807
  rule %r/:\p{Nd}+<[0-9a-z_.]+>/i, Num
616
808
  # imaginary numbers
617
- rule %r/(?:\p{Nd}+(?:_\p{Nd}+)*(?:\.\p{Nd}+(?:_\p{Nd}+)*)?|\.\p{Nd}+(?:_\p{Nd}+)*)(?:e[+-]?\p{Nd}+)?i(?![#{w}'\-])/i, Num::Float
618
- rule %r/(?:\p{Nd}+(?:_\p{Nd}+)*)?\.\p{Nd}+(?:_\p{Nd}+)*(?:e[+-]?\p{Nd}+)?/i, Num::Float
809
+ rule %r/(?:\p{Nd}+(?:_\p{Nd}+)*(?:\.\p{Nd}+(?:_\p{Nd}+)*)?|#{leading_dot}\p{Nd}+(?:_\p{Nd}+)*)(?:e[+-]?\p{Nd}+)?i(?![#{w}'\-])/i, Num::Float
810
+ rule %r/(?:\p{Nd}+(?:_\p{Nd}+)*\.|#{leading_dot})\p{Nd}+(?:_\p{Nd}+)*(?:e[+-]?\p{Nd}+)?/i, Num::Float
619
811
  rule %r/\p{Nd}+(?:_\p{Nd}+)*e[+-]?\p{Nd}+/i, Num::Float
620
812
  rule %r/\p{Nd}+(?:_\p{Nd}+)*/, Num::Integer
621
- # rational and complex literals: <1/3>, <1+2i>
622
- rule %r/<[-+]?\p{Nd}+\/\p{Nd}+>/, Num
623
- rule %r/<[-+]?[\p{Nd}.]+[-+][\p{Nd}.]+i>/, Num
813
+ # the other characters that are numbers: ⅒, ½, Ⅷ
814
+ rule %r/[\p{No}\p{Nl}&&[^#{sup}]]/, Num
815
+ # rational and complex literals: <1/3>, <0x01/0x02>, <1+2i>, <NaN+0i>
816
+ rule %r/<[-+]?(?:0[xob][0-9A-Fa-f]+|\p{Nd}+)\/(?:0[xob][0-9A-Fa-f]+|\p{Nd}+)[ \t]*>/, Num
817
+ rule %r/<[-+]?#{angle_number}[-+]#{angle_number}\\?i[ \t]*>/, Num
624
818
 
625
819
  # --- regex declarations, only when a name or a block follows
626
820
  rule %r/#{not_a_regex_declarator}(regex|token|rule)(\p{Space}+)(#{ident}:sym)/ do
@@ -641,40 +835,53 @@ module Rouge
641
835
  # A bracket can be repeated to make a longer delimiter (q<< >>);
642
836
  # other characters can not, so qq|| is an empty string.
643
837
  # and heredocs (q:to/END/). Before the keyword and builtin rules,
644
- # which would otherwise take q for a word.
645
- rule %r/#{quote_start}(qq|q|Q)[a-zA-Z]?(?:(#{adverbs})\p{Space}*|#{before_delimiter})((#{OPEN_BRACKET})\4*|[^0-9a-zA-Z:\p{Space}=,;)])/ do |m|
838
+ # which would otherwise take q for a word. With a parenthesis right
839
+ # after it, q is a routine that is being called: q(1)
840
+ rule %r/#{quote_start}(qq|q|Q)(ww|[a-zA-Z])?(?!\()(?:(#{adverbs})\p{Space}*|#{before_delimiter})((#{OPEN_BRACKET})\5*|#{quote_delimiter.(3)})/ do |m|
646
841
  opening = m[0]
647
- adverbs = m[2].to_s
648
- # qq strings (and the :qq / :c adverbs) interpolate
649
- interpolate = m[1] == 'qq' || adverbs.match?(/:(?:qq|c)\b/)
842
+ # (not called "adverbs": that is the pattern above, which the
843
+ # rules that are built later still need)
844
+ given = m[3].to_s
845
+ # qq strings interpolate, and so do the others as far as their
846
+ # adverbs say: q:s/$scalars only/, Qc/{closures} only/
847
+ interpolations = interpolations_of(m[1], m[2], given)
650
848
  # nothing is special in Q, not even a backslash: Q/\/
651
- body, closing = scan_delimited(m, m[3], escapes: m[1] != 'Q')
849
+ body, closing = scan_delimited(m, m[4], escapes: m[1] != 'Q')
652
850
 
653
- if adverbs.match?(/:to\b/)
851
+ if given.match?(HEREDOC_ADVERB)
654
852
  token Str, opening + body + closing
655
- lex_heredocs(m, body, interpolate)
656
- elsif interpolate
853
+ lex_heredocs(m, body, interpolations)
854
+ elsif interpolations.empty?
855
+ token Str, opening + body + closing
856
+ else
657
857
  token Str, opening
658
- sublex body, :interpolated
858
+ lex_quoted(body, interpolations)
659
859
  token Str, closing
660
- else
661
- token Str, opening + body + closing
662
860
  end
663
861
  end
664
862
 
665
863
  # --- regexes: m/x/, rx{x}, and with adverbs m:i/x/
666
- rule %r/#{quote_start}(?:m|ms|rx)(?:#{adverbs}\p{Space}*|#{before_delimiter})((#{OPEN_BRACKET})\2*|[^#{w}:\p{Space}=,;)])/ do |m|
667
- opener = m[1]
668
- token Str::Regex
864
+ rule %r/#{quote_start}(ms|m|rx)(?!\()(?:(#{regex_adverbs})\p{Space}*|#{before_delimiter})((#{OPEN_BRACKET})\4*|#{quote_delimiter.(2)})/ do |m|
865
+ opener = m[3]
866
+ lex_regex_opening(m[0], m[1], opener)
669
867
  lex_regex_part(m, opener, :regex_body)
670
868
  end
671
869
  # substitution and transliteration: s/a/b/, S{a}{b}, s:2nd/a/b/, tr/a-z/A-Z/
672
- # (a pattern can not be empty, so S|| and S%% are operators)
870
+ # (a pattern can not be empty, so S|| and S%% are operators, and so
871
+ # is S!~~)
673
872
  rule %r/#{quote_start}(ss|s|SS|S|tr|TR)(?=\p{Space}*:!?[#{w}])\p{Space}*(?::!?[#{w}\-]+(?:\([^)\n]*\))?\p{Space}*)+((#{OPEN_BRACKET})\3*|[^#{w}:\p{Space}$@%&=,;)])/ do |m|
674
873
  lex_substitution(m, m[0], m[1], m[2])
675
874
  end
676
- rule %r/#{quote_start}(ss|s|SS|S|tr|TR)#{before_delimiter}(([{(\[])\3*|([\/|!^~@%])(?!\4))/ do |m|
677
- lex_substitution(m, m[0], m[1], m[2])
875
+ rule %r/#{quote_start}(ss|s|SS|S|tr|TR)(?!!~~|\()#{before_delimiter}(([{(\[])\3*|([\/|!^~@%])(?!\4))/ do |m|
876
+ opening, kind, opener = m[0], m[1], m[2]
877
+ if opener.start_with?('[') && !substitution_follows?(m, opener)
878
+ # an index into something called s: s[0]
879
+ token Name, kind
880
+ token Text::Whitespace, opening[kind.length...-opener.length]
881
+ token Punctuation, opener
882
+ else
883
+ lex_substitution(m, opening, kind, opener)
884
+ end
678
885
  end
679
886
  # --- curly and corner quotes: ‘raw’ (or ‚this‘, or ’this‘),
680
887
  # “interpolating” (or „this“, or ”this“), 「no escapes」
@@ -687,15 +894,20 @@ module Rouge
687
894
  rule %r/#{after_dot}#{word_match.(method_builtins)}/, Name::Builtin
688
895
  rule %r/#{after_dot}#{ident}/, Name::Function
689
896
  # the key of a pair is a plain word, whatever it spells: name => 1
690
- rule %r/(?<!#{ident_char})#{qualified_ident}(?=\p{Space}*=>)/, Name
897
+ # (but Z=> between two terms is the zip operator)
898
+ rule %r/#{not_in_word}(?![RXZ]=>(?<=\p{Space}...))#{qualified_ident}(?=\p{Space}*=>)/, Name
691
899
  # operators used by name: infix:<+>(1, 2)
692
- rule %r/(?<!#{ident_char})#{op_categories}#{op_name_suffix}/, Name::Function
900
+ rule %r/#{not_in_word}#{op_categories}#{op_name_suffix}/, Name::Function
693
901
  # traits: is rw, is copy, is export
694
- rule %r/(?<!#{ident_char})(is)(\p{Space}+)#{word_match.(TRAITS)}/ do
902
+ rule %r/#{not_in_word}(is)(\p{Space}+)#{word_match.(TRAITS)}/ do
903
+ groups Keyword, Text::Whitespace, Keyword
904
+ end
905
+ # phasers as traits: will leave { ... }
906
+ rule %r/#{not_in_word}(will)(\p{Space}+)#{word_match.(WILL_TRAITS)}/ do
695
907
  groups Keyword, Text::Whitespace, Keyword
696
908
  end
697
909
  # user-defined types: "is Foo", "of Foo", "--> Foo"
698
- rule %r/(?<!#{ident_char})(is|does|of|returns|handles|trusts|hides)(\p{Space}+)#{not_a_builtin_type}(#{type_name}#{smiley}?)(?!#{ident_end})/ do
910
+ rule %r/#{not_in_word}(is|does|of|returns|handles|trusts|hides)(\p{Space}+)#{not_a_builtin_type}(#{type_name}#{smiley}?)(?!#{ident_end})/ do
699
911
  groups Keyword, Text::Whitespace, Name::Class
700
912
  end
701
913
  rule %r/(-->)(\p{Space}*)#{not_a_builtin_type}(#{type_name}#{smiley}?)(?!#{ident_end})/ do
@@ -723,6 +935,10 @@ module Rouge
723
935
  end
724
936
 
725
937
  # --- keywords and other reserved words
938
+ # the label of the loop to leave or to go on with: next OUTER
939
+ rule %r/#{word_match.(%w(next last redo))}(\p{Space}+)([A-Z][A-Z0-9_]*)(?=\p{Space}*(?:[;}]|if|unless|when|$))/ do
940
+ groups Keyword, Text::Whitespace, Name::Label
941
+ end
726
942
  rule %r/#{word_match.(DECLARATORS)}/, Keyword::Declaration
727
943
  rule %r/#{word_match.(NAMESPACE_KEYWORDS)}/, Keyword::Namespace
728
944
  rule %r/#{word_match.(KEYWORDS)}/, Keyword
@@ -730,10 +946,11 @@ module Rouge
730
946
  rule %r/#{word_match.(%w(self))}/, Name::Builtin::Pseudo
731
947
  rule %r/#{word_match.(CONSTANTS)}/, Name::Constant
732
948
  rule %r/[∞∅]/, Name::Constant
733
- # version literals: v6.d, v1.2.3, v1.2+
734
- rule %r/(?<!#{ident_char})v\p{Nd}+(?:\.(?:\p{Nd}+|\*|[a-z]))*\+?(?!#{ident_end})/, Num
735
- # meta operators: Z+, X~, R-, Z=>, Rcmp, and x=, xx=
736
- rule %r/#{nw}[RXZ](?:\*\*|\/\/|&&|\|\||<=>|=>|==|!=|<=|>=|~~|[-+*\/%~,&|^?<>]|(?:cmp|eq|ne|lt|gt|le|ge|leg|eqv|min|max|div|mod|and|or|xor|x|xx)(?![#{w}'\-]))/, Operator
949
+ # version literals: v6.d, v1.2.3, v1.2+, v6.e.PREVIEW
950
+ rule %r/#{not_in_word}v\p{Nd}+(?:\.(?:\p{Nd}+|\*|[a-z]|[A-Z]+)(?!#{ident_end}))*\+?(?!#{ident_end})/, Num
951
+ # meta operators: Z+, X~, R-, Z=>, Rcmp, S!~~, and x=, xx=
952
+ rule %r/#{nw}S(?:!~~|~~|&&|\|\||\^\^|%%|\/\/|==|!=|&(?=\p{Space})|xx?(?![#{w}'\-]))/, Operator
953
+ rule %r/#{nw}[RXZ](?:!~~|\*\*|\/\/|&&|\|\||<=>|=>|==|!=|<=|>=|~~|[-+*\/%~,&|^?<>]|(?:cmp|eq|ne|lt|gt|le|ge|leg|eqv|min|max|div|mod|and|or|xor|x|xx)(?![#{w}'\-]))/, Operator
737
954
  rule %r/#{nw}(?:xx?|min|max)=(?![=~>])/, Operator
738
955
  rule %r/#{word_match.(WORD_OPERATORS, '(?!\()')}/, Operator::Word
739
956
 
@@ -743,34 +960,43 @@ module Rouge
743
960
  rule %r/#{word_match.(BUILTIN_CLASSES, "#{smiley}?")}/, Keyword::Type
744
961
  rule %r/#{word_match.(BUILTINS)}/, Name::Builtin
745
962
  # a user-defined type: with a smiley, or constraining a variable
746
- rule %r/(?<!#{ident_char})#{type_name}#{smiley}/, Name::Class
747
- rule %r/(?<!#{ident_char})#{type_name}(?=\p{Space}+[$@%&\\])/, Name::Class
963
+ rule %r/#{not_in_word}#{type_name}#{smiley}/, Name::Class
964
+ rule %r/#{not_in_word}#{type_name}(?=\p{Space}+[$@%&\\])/, Name::Class
748
965
  rule %r/(?<=[#{w}>])#{smiley}/, Keyword::Type
749
966
 
750
967
  # --- operators that could be mistaken for quotes
751
- # hyper operators: »+«, >>+<<, +« (prefix), »++ and ».say (postfix)
752
- rule %r/(?:«|»|<<|>>)[-+*\/%~=!&|^?<>]+(?:«|»|<<|>>)/, Operator
968
+ # hyper operators: »+«, >>+<<, »⋅«, +« (prefix), »++ and ».say (postfix)
969
+ rule %r/(?:«|»|<<|>>)[-+*\/%~=!&|^?<>\p{Sm}]+(?:«|»|<<|>>)/, Operator
753
970
  rule %r/[-+*\/%~!?|^]+«/, Operator
754
971
  # an ASCII prefix hyper, when its operand follows: -<< (3, 2, 1)
755
- rule %r/[-+~!?|^]<<(?=\p{Space}*[$@%(\[])/, Operator
972
+ rule %r/[-+~!?|^]<<(?=\p{Space}*[$@%(\[<])/, Operator
756
973
  # ... and around a bracketed operator: «[op]«, <<[op]>>. With a
757
974
  # variable inside it is a subscript: @rows>>[$i]>>.chars
758
975
  rule %r/(?:«|»|<<|>>)\[[^\]\n$@%]+\](?:«|»|<<|>>)/, Operator
759
976
  rule %r/(?:»|>>)(?=\.[#{w}^?!(]|\+\+|--|[\[{<(])/, Operator
760
977
  # reduction operators: [+], [\*], [<=], [max], [Z+]
761
978
  rule %r/(?<![#{w}\])}>'\-])\[\\?(?:[RXZ]?[-+*\/%~=<>!&|^?,]+|[RXZ]?(?:min|max|gcd|lcm|and|or|xor|x|xx|cmp|eq|ne|lt|gt|le|ge|leg|eqv))\]/, Operator
979
+ # a prefix operator in front of a word list: ~<a b c>
980
+ rule %r/[~+?](?=<[^\p{Space}=<>{};()][^<>{};()\n]*>)/, Operator
981
+ # the empty subscript: @a[0]<>
982
+ rule %r/<>(?<=[#{w}\])}]<>)/, Str
762
983
  # quote-like words with interpolation: «a $b» and <<a $b>>. After
763
984
  # an operator or a term these are hyper operators, not quotes,
764
985
  # and so they are in [«] and [<<], which name an operator.
765
986
  rule %r/(?<![#{w}\])}>+\-*\/%~!?^|&.])«(?!\])/, Str::Double, :ww_guillemets
766
987
  rule %r/(?<![#{w}\])}>])<<(?![=\]])/, Str::Double, :ww_angles
767
- rule %r/(?!<->)<[^\p{Space}=<>{};()](?:[^<>{};()]*[^\p{Space}<>{};()])?>/, Str
988
+ rule %r/(?!<->)<(?:\\[<>]|[^\p{Space}=<>{};()])(?:(?:\\[<>]|[^<>{};()])*(?:\\[<>]|[^\p{Space}<>{};()]))?>/, Str
768
989
 
769
990
  # --- labels, pairs and operators
770
991
  # loop labels: OUTER: for ...
771
992
  rule %r/#{nw}([A-Z][A-Z0-9_]*)(:)(?=\p{Space}*(?:for|while|until|loop|repeat|given|if|unless|do|\{|$))/ do
772
993
  groups Name::Label, Punctuation
773
994
  end
995
+ # a colon-pair with interpolating words for its value: :name«$x»
996
+ rule %r/(:!?)(#{ident})(«|<<(?![<=]))/ do |m|
997
+ groups Punctuation, Name::Attribute, Str::Double
998
+ push(m[3] == '«' ? :ww_guillemets : :ww_angles)
999
+ end
774
1000
  # colon-pairs and adverbs: :name, :!flag, :name(...), :2nd
775
1001
  rule %r/(:!?)(#{ident})/ do
776
1002
  groups Punctuation, Name::Attribute
@@ -849,13 +1075,35 @@ module Rouge
849
1075
  end
850
1076
  # character classes: <[a..z]>, <-[\p{Nd}] + [_]>
851
1077
  rule %r/<(?:[-+!?.]\p{Space}*)?\[(?:\\.|[^\]\\])*\](?:\p{Space}*[-+]\p{Space}*(?:\[(?:\\.|[^\]\\])*\]|:?[#{w}\-]+))*\p{Space}*>/m, Str::Regex
852
- # unicode properties: <:Lu>, <+:L>, <:L + :N>, <:L - [a]>
853
- rule %r/<[-+!?.]?\p{Space}*:[#{w}\-]+(?:\([^)\n]*\))?(?:\p{Space}*[-+]\p{Space}*(?::[#{w}\-]+|\[(?:\\.|[^\]\\])*\]))*\p{Space}*>/m, Name::Builtin
1078
+ # unicode properties: <:Lu>, <+:L>, <:L + :N>, <:L - [a]>, <:bc<L>>
1079
+ rule %r/<[-+!?.]?\p{Space}*:[#{w}\-]+(?:\([^)\n]*\)|<[^<>\n]*>)?(?:\p{Space}*[-+]\p{Space}*(?::[#{w}\-]+|\[(?:\\.|[^\]\\])*\]))*\p{Space}*>/m, Name::Builtin
854
1080
  # named assertions: <foo>, <.foo>, <?before ...>, <name=rule>
855
1081
  rule %r/(<)([?!.+-]?)(\p{Space}*)(#{ident})(=)(#{ident})(>)/ do
856
1082
  groups Punctuation, Punctuation, Text::Whitespace, Name::Variable,
857
1083
  Operator, Name::Function, Punctuation
858
1084
  end
1085
+ # ... with arguments, which are code: <expr(3)>, <term=.expr(3)>,
1086
+ # <indent: 2, 3>
1087
+ rule %r/(<)([?!.+-]?)(\p{Space}*)(?:(#{ident})(=)(\.?))?(#{ident})(#{arguments})/ do |m|
1088
+ call = m[8]
1089
+ token Punctuation, m[1]
1090
+ token Punctuation, m[2]
1091
+ token Text::Whitespace, m[3]
1092
+ token Name::Variable, m[4]
1093
+ token Operator, m[5]
1094
+ token Punctuation, m[6]
1095
+ token REGEX_BUILTINS.include?(m[7]) ? Name::Builtin : Name::Function, m[7]
1096
+ sublex call
1097
+ end
1098
+ rule %r/(<)([?!.+-]?)(\p{Space}*)(#{ident})(:)(\p{Space}[^<>\n]*)(?=>)/ do |m|
1099
+ call = m[6]
1100
+ token Punctuation, m[1]
1101
+ token Punctuation, m[2]
1102
+ token Text::Whitespace, m[3]
1103
+ token Name::Function, m[4]
1104
+ token Punctuation, m[5]
1105
+ sublex call
1106
+ end
859
1107
  rule %r/(<)([?!.+-]?)(\p{Space}*)#{word_match.(REGEX_BUILTINS)}(>)?/ do
860
1108
  groups Punctuation, Punctuation, Text::Whitespace, Name::Builtin, Punctuation
861
1109
  end
@@ -870,7 +1118,9 @@ module Rouge
870
1118
  end
871
1119
  rule %r/\$<[#{w}'\-]+>/, Name::Variable
872
1120
  rule %r/\$\p{Nd}+/, Name::Variable
873
- rule %r/[$@][.^:?=!~*]?#{qualified_ident}#{angle_subscripts}/, Name::Variable
1121
+ rule %r/([$@][.^:?=!~*]?#{qualified_ident})(#{angle_subscripts})/ do |m|
1122
+ lex_variable(Name::Variable, m[1], m[2])
1123
+ end
874
1124
  # literals
875
1125
  rule %r/'(?:\\.|[^'\\])*'/m, Str::Single
876
1126
  rule %r/[‘‚’][^‘’\n]*[’‘]/, Str::Single
@@ -911,36 +1161,113 @@ module Rouge
911
1161
 
912
1162
  # Pod documentation blocks
913
1163
  state :pod_body do
914
- rule %r/^(\p{Space}*)(=head\p{Nd}*)(.*)/ do
915
- groups Text::Whitespace, Comment::Preproc, Generic::Heading
1164
+ # Blocks whose content is taken as it is: a "=begin" inside one
1165
+ # does not open a block, and the "=end" has to be indented like
1166
+ # the "=begin", so that an indented example does not end it.
1167
+ rule %r/^([ \t]*)(=begin)([ \t]+)(code|comment|input|output|data)(?![#{w}\-])([^\n]*\n)(.*?)(?=^\1=end[ \t]+\4(?![#{w}\-]))/m do |m|
1168
+ lex_pod_verbatim(m[4], m[5], m[6], [m[1], m[2], m[3]])
916
1169
  end
917
- rule %r/^(\p{Space}*)(=(?:begin|end|for|finish))([ \t]*)([#{w}]*)/ do
918
- groups Text::Whitespace, Comment::Preproc, Text::Whitespace, Name::Namespace
1170
+ # the same as a paragraph, which ends at a blank line or at the
1171
+ # next directive, and abbreviated: =code say 1;
1172
+ rule %r/^([ \t]*)(=for)([ \t]+)(code)(?![#{w}\-])([^\n]*\n?)(#{pod_paragraph})/ do |m|
1173
+ lex_pod_verbatim(m[4], m[5], m[6], [m[1], m[2], m[3]])
919
1174
  end
920
- rule %r/^(\p{Space}*)(=[A-Za-z][#{w}]*)/ do
921
- groups Text::Whitespace, Comment::Preproc
1175
+ rule %r/^([ \t]*)(=code)(?![#{w}\-])([ \t]*)(#{pod_paragraph})/ do |m|
1176
+ code = m[4]
1177
+ token Text::Whitespace, m[1]
1178
+ token Comment::Preproc, m[2]
1179
+ token Text::Whitespace, m[3]
1180
+ sublex code
922
1181
  end
923
- # formatting codes: B<bold>, I<italic>, C<code>, L<link>, ...
924
- rule %r/(B)(<)([^<>\n]*)(>)/ do
925
- groups Name::Decorator, Punctuation, Generic::Strong, Punctuation
1182
+
1183
+ rule %r/^([ \t]*)(=head\p{Nd}*)(.*)/ do
1184
+ groups Text::Whitespace, Comment::Preproc, Generic::Heading
926
1185
  end
927
- rule %r/(I)(<)([^<>\n]*)(>)/ do
928
- groups Name::Decorator, Punctuation, Generic::Emph, Punctuation
1186
+ rule %r/^([ \t]*)(=(?:begin|for))([ \t]*)([#{w}\-]*)/ do
1187
+ groups Text::Whitespace, Comment::Preproc, Text::Whitespace, Comment::Preproc
1188
+ push :pod_config
929
1189
  end
930
- rule %r/([CKTV])(<)([^<>\n]*)(>)/ do
931
- groups Name::Decorator, Punctuation, Str::Backtick, Punctuation
1190
+ rule %r/^([ \t]*)(=(?:end|finish))([ \t]*)([#{w}\-]*)/ do
1191
+ groups Text::Whitespace, Comment::Preproc, Text::Whitespace, Comment::Preproc
932
1192
  end
933
- rule %r/([ALENPRSUXZ])(<)([^<>\n]*)(>)/ do
934
- groups Name::Decorator, Punctuation, Comment::Multiline, Punctuation
1193
+ rule %r/^([ \t]*)(=[A-Za-z][#{w}]*)/ do
1194
+ groups Text::Whitespace, Comment::Preproc
935
1195
  end
936
- rule %r/[^\n]+?(?=[A-Z]<|$)/, Comment::Multiline
1196
+ mixin :pod_format_start
1197
+ rule %r/[^\n]+?(?=[A-Z][<«]|$)/, Comment::Multiline
937
1198
  rule %r/[A-Z]/, Comment::Multiline
938
1199
  rule %r/\n/, Text::Whitespace
939
1200
  end
940
1201
 
1202
+ # the options of a directive: :kind<Type>, :caption("x"), :!numbered
1203
+ state :pod_config_pairs do
1204
+ rule %r/[ \t]+/, Text::Whitespace
1205
+ rule %r/(:!?)([#{w}\-]+)(<[^>\n]*>|\([^)\n]*\)|\[[^\]\n]*\]|\{[^}\n]*\}|«[^»\n]*»)?/ do
1206
+ groups Punctuation, Name::Attribute, Str
1207
+ end
1208
+ end
1209
+
1210
+ state :pod_config do
1211
+ mixin :pod_config_pairs
1212
+ rule(//) { pop! }
1213
+ end
1214
+
1215
+ # the rest of the line that opens a block
1216
+ state :pod_config_line do
1217
+ mixin :pod_config_pairs
1218
+ rule %r/[^\n]+/, Comment::Multiline
1219
+ rule %r/\n/, Text::Whitespace
1220
+ end
1221
+
1222
+ # formatting codes: B<bold>, I<italic>, C<code>, L<link>, ... with
1223
+ # their other delimiters, C<< a > b >> and C«a > b»
1224
+ state :pod_format_start do
1225
+ rule %r/([A-Z])(<<|<|«)/ do |m|
1226
+ open_pod_format(m[1], m[2])
1227
+ end
1228
+ end
1229
+
1230
+ # The text of a formatting code. The codes nest, and one that is not
1231
+ # closed stops at the end of its paragraph.
1232
+ state :pod_format do
1233
+ rule %r/\n(?=[ \t]*(?:\n|\z|=[A-Za-z]))/ do
1234
+ token Text::Whitespace
1235
+ pop!(@pod_formats.length)
1236
+ @pod_formats.clear
1237
+ end
1238
+ rule %r/\n/, Text::Whitespace
1239
+ mixin :pod_format_start
1240
+ # angle brackets are counted, and so the first ">" does not close
1241
+ # C<%h<key>>; with the other delimiters they are plain text
1242
+ rule %r/</ do
1243
+ closer, text, = @pod_formats.last
1244
+ @pod_formats.last[2] += 1 if closer == '>'
1245
+ token text
1246
+ end
1247
+ rule %r/[>»]/ do |m|
1248
+ bracket = m[0]
1249
+ closer, text, depth = @pod_formats.last
1250
+ if closer == '>' && bracket == '>' && depth.positive?
1251
+ @pod_formats.last[2] -= 1
1252
+ token text, bracket
1253
+ elsif closer == bracket || (closer == '>>' && bracket == '>' && m.scan(/>/))
1254
+ token Punctuation, closer
1255
+ @pod_formats.pop
1256
+ pop!
1257
+ else
1258
+ token text, bracket
1259
+ end
1260
+ end
1261
+ rule %r/[^<>«»\nA-Z]+|[A-Z«]/ do
1262
+ token @pod_formats.last[1]
1263
+ end
1264
+ end
1265
+
941
1266
  # what can appear inside an interpolating string
942
1267
  state :interpolation do
943
- rule %r/\\(?:[abefnrt0"'\\$@%&{}<>«»]|[xXoOcCdD]\[[^\]\n]*\]|x[0-9a-fA-F]+)/, Str::Escape
1268
+ rule %r/\\(?:[abefnrt0"'\\$@%&{}<>«»]|[xXoOcCdD]\[[^\]\n]*\]|x[0-9a-fA-F]+)/ do
1269
+ token interpolates?(:b) ? Str::Escape : Str::Double
1270
+ end
944
1271
  rule %r/\\./m, Str::Double
945
1272
  # What interpolates is code, and is lexed as code, so that a
946
1273
  # variable looks the same inside a string as outside of it.
@@ -952,12 +1279,20 @@ module Rouge
952
1279
  # Each can be followed by subscripts and by method calls, which
953
1280
  # need their parentheses here: $x[0]<a>, $x.uc(), @a.sort().join(',')
954
1281
  rule %r/(?:[$@%&]#{arguments}#{postfix}*|\$(?:[*.!^?=~:]?#{qualified_ident}|[!\/&¢]|\p{Nd}+|<[^>\n]+>)#{postfix}*|[@%][*.!^?=~:]?#{qualified_ident}#{postfix}+|&#{ident}#{arguments}#{postfix}*)/ do |m|
955
- sublex m[0]
1282
+ if interpolates?(SIGIL_INTERPOLATIONS[m[0][0]])
1283
+ sublex m[0]
1284
+ else
1285
+ token Str::Double
1286
+ end
956
1287
  end
957
1288
  rule %r/\{/ do
958
- token Punctuation
959
- @brace_levels << 1
960
- push :embedded
1289
+ if interpolates?(:c)
1290
+ token Punctuation
1291
+ @brace_levels << 1
1292
+ push :embedded
1293
+ else
1294
+ token Str::Double
1295
+ end
961
1296
  end
962
1297
  end
963
1298
 
@@ -984,17 +1319,20 @@ module Rouge
984
1319
 
985
1320
  state :ww_guillemets do
986
1321
  rule %r/»/, Str::Double, :pop!
1322
+ # a word can not start with "#": it starts a comment
1323
+ rule %r/(?<=\p{Space})#.*/, Comment::Single
987
1324
  mixin :interpolation
988
- rule %r/[^»\\$@%&{]+/, Str::Double
989
- rule %r/[$@%&\\]/, Str::Double
1325
+ rule %r/[^»\\$@%&{#]+/, Str::Double
1326
+ rule %r/[$@%&\\#]/, Str::Double
990
1327
  end
991
1328
 
992
1329
  state :ww_angles do
993
1330
  rule %r/>>/, Str::Double, :pop!
1331
+ rule %r/(?<=\p{Space})#.*/, Comment::Single
994
1332
  mixin :interpolation
995
- rule %r/[^>\\$@%&{]+/, Str::Double
1333
+ rule %r/[^>\\$@%&{#]+/, Str::Double
996
1334
  # (a lone '>' does not close the quote)
997
- rule %r/[$@%&\\>]/, Str::Double
1335
+ rule %r/[$@%&\\>#]/, Str::Double
998
1336
  end
999
1337
  end
1000
1338
  end
@@ -2,6 +2,6 @@
2
2
 
3
3
  module Rouge
4
4
  module Raku
5
- VERSION = "0.1.0"
5
+ VERSION = "0.2.0"
6
6
  end
7
7
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: rouge-raku
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
4
+ version: 0.2.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Naoum Hankache
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2026-10-05 00:00:00.000000000 Z
11
+ date: 2026-10-10 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: rouge