buninu 0.6.0 → 0.7.0

package/apps/jsmdcui/tests/syntax-parity/NORMAL-HTML.md

@@ -0,0 +1,46 @@
+# Normal HTML follow-up
+
+Tested with the same unmodified micro highlighter oracle and upstream YAML as
+REPORT.md. See normal-html-report.json for counts and first differing positions.
+The full comparison.html was also passed to both engines as input, unchanged.
+
+The minimal reproduction is ordinary valid HTML:
+
+```html
+<div class="hello">Hello</div>
+<p>Next line</p>
+```
+
+With html.yaml (the definition selected by the .html filename), jmi ends the
+first line with a constant.string state and colors the entire second line as a
+string. This is a real nested-region state bug, not just a malformed generated
+sample. normal-html.html contains a complete document for manual reproduction.
+
+For the generated comparison.html, html.yaml produced 731354 differing character
+groups, including 438494 characters marked as strings only by jmi. This large
+one-line file amplifies the state issue; it is not a typical per-file error rate.
+The same full file matched with explicitly selected html4.yaml and html5.yaml,
+which have different rules and are not the default .html filename match.
+
+This check did not modify the highlighter or the syntax files.
+
+## After the nested-region fix
+
+The historical reproduction above is now covered by highlight-regions.test.js.
+The child scanner already resumes its parent and processes the rest of the line;
+returning that result directly avoids rescanning closing quotes as new openings.
+Both HTML and XML now return to the correct outer state, including when an
+attribute legitimately spans multiple lines. Original syntax YAML is unchanged.
+
+The current normal-html-report.json records the follow-up: simple HTML,
+two-attribute HTML and a complete normal document have no extra string-colored
+characters. Their remaining differences are local token groups, such as micro
+using symbol.tag where jmi recognizes an attribute as identifier. The current
+comparison.html still has local group differences and is not fully identical.
+It is regenerated from the latest results, so its total size differs from the
+historical input above; raw differing-character totals are not a fixed-corpus
+before/after benchmark.
+
+The 158-syntax corpus now has 130 identical samples and 28 with differences.
+XML and MSBuild samples match completely. Mixed POSIX character classes are
+also translated; default/email, mail and privoxy-action samples now match.

package/apps/jsmdcui/tests/syntax-parity/README.md

@@ -0,0 +1,85 @@
+# micro syntax parity fixtures
+
+This directory contains one lexical sample for each of **158 syntax definitions**.
+The current report uses upstream micro YAML for both engines, including
+JavaScript, Python and Markdown. For the first run those three local files were
+backed up, temporarily replaced and restored byte-for-byte after comparison.
+`--upstream-syntax` reproduces that comparison without modifying any syntax files.
+Without this flag, the comparator tests only byte-identical local definitions
+and excludes JavaScript, Python and Markdown; included definitions then come
+from each engine's own tree.
+
+- [REPORT.md](REPORT.md): counts and first differences, with sample links.
+- `comparison.html` (generated locally, not included): side-by-side colors using
+  the same Monokai palette; hover over a span for its exact group. No external
+  assets required.
+- [report.json](report.json): machine-readable per-language results.
+- `samples/*.txt`: generated rule witnesses, with hand-written Dockerfile and
+  MSBuild examples. They are lexical test fragments, **not generally valid
+  programs**. Region boundaries, escaping, and overlapping patterns are included.
+
+The oracle invokes the actual, unmodified upstream `pkg/highlight` Go package,
+including its YAML parser and include resolution. No Go regex approximation is
+used for the reference results. This is a highlighter comparison, not a terminal
+screenshot comparison or a test of automatic filetype detection. The sample
+filenames deliberately do not guess extensions: select the filetype shown in
+the report when opening them in an editor (`set filetype dockerfile`, for example).
+
+All samples have non-default groups in both engines. Equal results mean equal
+per-character syntax groups **for these samples only**, not full language parity.
+The HTML is a viewing aid, not a substitute for the group-level comparison.
+
+## Generate the comparison HTML and reports
+
+Install Go 1.24 or newer and Bun with scoped RegExp modifier support. Start in
+this directory. Point `MICRO_SOURCE` at an existing upstream micro checkout:
+
+```sh
+export MICRO_SOURCE=/absolute/path/to/go-micro
+export PARITY_BUILD_DIR="$(mktemp -d)"
+cp go.mod "$PARITY_BUILD_DIR/oracle.mod"
+go mod edit -modfile="$PARITY_BUILD_DIR/oracle.mod" \
+  -replace="github.com/micro-editor/micro/v2=$MICRO_SOURCE"
+go build -mod=mod -modfile="$PARITY_BUILD_DIR/oracle.mod" \
+  -o "$PARITY_BUILD_DIR/oracle" oracle.go
+bun compare.mjs "$MICRO_SOURCE" "$PARITY_BUILD_DIR/oracle" --upstream-syntax
+```
+
+The final `bun compare.mjs` command writes `comparison.html`, `REPORT.md` and
+`report.json` into this directory. Open `comparison.html` in a browser and expand
+a language to compare micro (left) with jmi (right). Once the oracle executable
+has been built, rerun just that final command to regenerate the page. The HTML
+is intentionally not checked in and has no Git ignore rule; remove it after
+viewing if you do not want it included in a later commit.
+
+Go downloads test-only dependencies to its module cache. It is not a jsmdcui
+runtime dependency; the checkout's module file and upstream sources remain
+unchanged. The reference commit is recorded in the report. Re-running rewrites
+only the reports; it keeps the checked-in samples. `--generate` regenerates
+lexical witnesses and overwrites hand-written samples, so normal comparisons
+should omit it.
+
+Differences are reported rather than hidden or automatically fixed. In
+particular, a regex compiling successfully does not establish equivalence of
+nested region handling, skip masking, or rule ordering.
+
+## Scanner safety checks
+
+Run `bun test tests/highlight-regions.test.js tests/go-regexp.test.js` from the
+jsmdcui root. These cover zero-width cycles, Unicode empty matches, NUL, isolated
+surrogates, cyclic definitions/state, and a 200,003-character line whose final
+keyword must still be highlighted. Region transitions are iterative; there is
+no input line-length cutoff. Malformed definitions are limited to 128 nested
+rule levels. A repeated state at the same position advances by one Unicode
+character and continues highlighting. If malformed states keep creating fresh
+objects, 256 transitions without advancing reset the unfinished region for that
+line instead of carrying it forward.
+
+A manual stability run also processed Vue 3.5.13's `vue.global.prod.js`
+(157,924 bytes, longest line 141,082 UTF-16 units) in about 87 ms on the test
+host, without a stack overflow or truncation. This is not proof of exact Vue
+syntax parity. No downloaded Vue bundle is included in the repository.
+
+These guards prevent scanner loops and stack exhaustion. They cannot interrupt
+a single pathological native JavaScript RegExp execution; arbitrary untrusted
+regexes would need worker/process isolation for a hard execution-time limit.

package/apps/jsmdcui/tests/syntax-parity/REPORT.md

@@ -0,0 +1,328 @@
+# micro / jmi syntax parity
+
+Upstream commit: 6cf68963a6c0630b696c0e56ae61f733ab7129c0
+
+Syntax source: upstream.
+
+158 byte-identical YAML files tested; 0 changed/missing files excluded.
+130 match; 28 differ; 0 lack non-default groups in one or both engines.
+
+Comparison is by syntax group for every character, not theme RGB values.
+These generated lexical samples exercise rule witnesses, not complete valid programs or exhaustive language coverage.
+
+| Syntax | micro colored chars | jmi colored chars | Different chars |
+|---|---:|---:|---:|
+| [powershell](samples/PowerShell.txt) | 297 | 297 | 1 |
+| [ada](samples/ada.txt) | 132 | 132 | 0 |
+| [apacheconf](samples/apacheconf.txt) | 566 | 566 | 0 |
+| [ino](samples/arduino.txt) | 300 | 300 | 0 |
+| [asciidoc](samples/asciidoc.txt) | 123 | 123 | 0 |
+| [asm](samples/asm.txt) | 450 | 450 | 0 |
+| [ATS](samples/ats.txt) | 365 | 367 | 2 |
+| [awk](samples/awk.txt) | 179 | 179 | 0 |
+| [B](samples/b.txt) | 185 | 185 | 0 |
+| [batch](samples/bat.txt) | 182 | 182 | 0 |
+| [c](samples/c.txt) | 283 | 283 | 0 |
+| [caddyfile](samples/caddyfile.txt) | 97 | 97 | 0 |
+| [cake](samples/cake.txt) | 76 | 76 | 0 |
+| [clojure](samples/clojure.txt) | 91 | 91 | 0 |
+| [cmake](samples/cmake.txt) | 117 | 117 | 0 |
+| [coffeescript](samples/coffeescript.txt) | 201 | 201 | 0 |
+| [colortest](samples/colortest.txt) | 99 | 99 | 0 |
+| [conky](samples/conky.txt) | 37 | 37 | 0 |
+| [c++](samples/cpp.txt) | 383 | 383 | 0 |
+| [crontab](samples/crontab.txt) | 62 | 62 | 0 |
+| [crystal](samples/crystal.txt) | 252 | 250 | 2 |
+| [csharp](samples/csharp.txt) | 213 | 213 | 0 |
+| [css](samples/css.txt) | 187 | 187 | 0 |
+| [csharp-script](samples/csx.txt) | 71 | 71 | 1 |
+| [cuda](samples/cuda.txt) | 262 | 262 | 0 |
+| [cython](samples/cython.txt) | 142 | 142 | 0 |
+| [d](samples/d.txt) | 452 | 452 | 4 |
+| [dart](samples/dart.txt) | 198 | 198 | 0 |
+| [unknown](samples/default.txt) | 37 | 37 | 0 |
+| [dockerfile](samples/dockerfile.txt) | 167 | 167 | 0 |
+| [dot](samples/dot.txt) | 125 | 125 | 0 |
+| [elixir](samples/elixir.txt) | 91 | 91 | 0 |
+| [elm](samples/elm.txt) | 129 | 129 | 0 |
+| [erb](samples/erb.txt) | 79 | 79 | 0 |
+| [erlang](samples/erlang.txt) | 128 | 128 | 0 |
+| [fish](samples/fish.txt) | 97 | 97 | 0 |
+| [forth](samples/forth.txt) | 125 | 125 | 0 |
+| [fortran](samples/fortran.txt) | 268 | 268 | 0 |
+| [freebsd-kernel](samples/freebsd-kernel.txt) | 70 | 70 | 0 |
+| [fsharp](samples/fsharp.txt) | 129 | 129 | 0 |
+| [gdscript](samples/gdscript.txt) | 187 | 187 | 0 |
+| [gemini](samples/gemini.txt) | 119 | 119 | 0 |
+| [ebuild](samples/gentoo-ebuild.txt) | 146 | 146 | 0 |
+| [etc-portage](samples/gentoo-etc-portage.txt) | 103 | 103 | 0 |
+| [git-commit](samples/git-commit.txt) | 146 | 146 | 0 |
+| [git-config](samples/git-config.txt) | 51 | 51 | 0 |
+| [git-rebase-todo](samples/git-rebase-todo.txt) | 29 | 29 | 0 |
+| [gleam](samples/gleam.txt) | 239 | 299 | 63 |
+| [glsl](samples/glsl.txt) | 145 | 145 | 0 |
+| [gnuplot](samples/gnuplot.txt) | 44 | 44 | 0 |
+| [go](samples/go.txt) | 213 | 213 | 0 |
+| [godoc](samples/godoc.txt) | 114 | 114 | 0 |
+| [golo](samples/golo.txt) | 336 | 336 | 0 |
+| [gomod](samples/gomod.txt) | 138 | 138 | 0 |
+| [graphql](samples/graphql.txt) | 87 | 87 | 0 |
+| [groff](samples/groff.txt) | 71 | 71 | 1 |
+| [groovy](samples/groovy.txt) | 333 | 333 | 2 |
+| [haml](samples/haml.txt) | 43 | 43 | 0 |
+| [hare](samples/hare.txt) | 179 | 179 | 1 |
+| [haskell](samples/haskell.txt) | 135 | 135 | 0 |
+| [hc](samples/hc.txt) | 186 | 186 | 0 |
+| [html](samples/html.txt) | 170 | 181 | 12 |
+| [html4](samples/html4.txt) | 11 | 19 | 9 |
+| [html5](samples/html5.txt) | 11 | 19 | 9 |
+| [ini](samples/ini.txt) | 86 | 86 | 0 |
+| [inputrc](samples/inputrc.txt) | 37 | 37 | 0 |
+| [java](samples/java.txt) | 150 | 150 | 0 |
+| [javascript](samples/javascript.txt) | 296 | 296 | 2 |
+| [jinja2](samples/jinja2.txt) | 42 | 79 | 37 |
+| [json](samples/json.txt) | 178 | 178 | 0 |
+| [jsonnet](samples/jsonnet.txt) | 249 | 249 | 4 |
+| [julia](samples/julia.txt) | 159 | 159 | 0 |
+| [justfile](samples/justfile.txt) | 146 | 146 | 0 |
+| [keymap](samples/keymap.txt) | 62 | 62 | 0 |
+| [kickstart](samples/kickstart.txt) | 53 | 53 | 0 |
+| [kotlin](samples/kotlin.txt) | 220 | 220 | 0 |
+| [kvlang](samples/kvlang.txt) | 256 | 256 | 0 |
+| [ledger](samples/ledger.txt) | 42 | 42 | 0 |
+| [lfe](samples/lfe.txt) | 51 | 51 | 0 |
+| [lilypond](samples/lilypond.txt) | 68 | 68 | 0 |
+| [lisp](samples/lisp.txt) | 32 | 32 | 0 |
+| [log](samples/log.txt) | 459 | 459 | 0 |
+| [lua](samples/lua.txt) | 490 | 490 | 5 |
+| [mail](samples/mail.txt) | 70 | 70 | 0 |
+| [makefile](samples/makefile.txt) | 142 | 142 | 0 |
+| [man](samples/man.txt) | 20 | 20 | 0 |
+| [markdown](samples/markdown.txt) | 159 | 159 | 0 |
+| [mc](samples/mc.txt) | 62 | 62 | 0 |
+| [meson](samples/meson.txt) | 226 | 226 | 0 |
+| [micro](samples/micro.txt) | 200 | 200 | 0 |
+| [mpd](samples/mpdconf.txt) | 53 | 53 | 0 |
+| [msbuild](samples/msbuild.txt) | 164 | 164 | 0 |
+| [nanorc](samples/nanorc.txt) | 78 | 78 | 0 |
+| [nftables](samples/nftables.txt) | 96 | 96 | 1 |
+| [nginx](samples/nginx.txt) | 68 | 68 | 0 |
+| [nim](samples/nim.txt) | 113 | 113 | 0 |
+| [nix](samples/nix.txt) | 123 | 123 | 0 |
+| [nu](samples/nu.txt) | 773 | 773 | 0 |
+| [objective-c](samples/objc.txt) | 265 | 265 | 12 |
+| [ocaml](samples/ocaml.txt) | 104 | 104 | 0 |
+| [octave](samples/octave.txt) | 206 | 206 | 2 |
+| [odin](samples/odin.txt) | 217 | 217 | 0 |
+| [pascal](samples/pascal.txt) | 169 | 169 | 2 |
+| [patch](samples/patch.txt) | 22 | 22 | 0 |
+| [peg](samples/peg.txt) | 52 | 52 | 0 |
+| [perl](samples/perl.txt) | 239 | 239 | 0 |
+| [php](samples/php.txt) | 283 | 287 | 16 |
+| [pc](samples/pkg-config.txt) | 32 | 32 | 0 |
+| [po](samples/po.txt) | 34 | 34 | 0 |
+| [pony](samples/pony.txt) | 156 | 156 | 0 |
+| [pov](samples/pov.txt) | 126 | 126 | 0 |
+| [privoxy-action](samples/privoxy-action.txt) | 56 | 56 | 0 |
+| [privoxy-config](samples/privoxy-config.txt) | 44 | 44 | 0 |
+| [privoxy-filter](samples/privoxy-filter.txt) | 34 | 34 | 0 |
+| [proto](samples/proto.txt) | 152 | 152 | 0 |
+| [prql](samples/prql.txt) | 293 | 293 | 12 |
+| [puppet](samples/puppet.txt) | 86 | 86 | 0 |
+| [python2](samples/python2.txt) | 148 | 148 | 0 |
+| [python](samples/python3.txt) | 174 | 174 | 0 |
+| [r](samples/r.txt) | 87 | 87 | 0 |
+| [raku](samples/raku.txt) | 120 | 120 | 0 |
+| [rst](samples/reST.txt) | 42 | 42 | 0 |
+| [renpy](samples/renpy.txt) | 56 | 56 | 0 |
+| [rpmspec](samples/rpmspec.txt) | 191 | 191 | 0 |
+| [ruby](samples/ruby.txt) | 285 | 285 | 0 |
+| [rust](samples/rust.txt) | 250 | 250 | 2 |
+| [sage](samples/sage.txt) | 148 | 148 | 0 |
+| [OpenSCAD](samples/scad.txt) | 173 | 173 | 0 |
+| [scala](samples/scala.txt) | 152 | 152 | 0 |
+| [sed](samples/sed.txt) | 27 | 27 | 0 |
+| [shell](samples/sh.txt) | 99 | 99 | 0 |
+| [salt](samples/sls.txt) | 48 | 48 | 0 |
+| [smalltalk](samples/smalltalk.txt) | 239 | 239 | 0 |
+| [solidity](samples/solidity.txt) | 230 | 230 | 0 |
+| [sql](samples/sql.txt) | 169 | 169 | 0 |
+| [stata](samples/stata.txt) | 244 | 246 | 10 |
+| [svelte](samples/svelte.txt) | 46 | 46 | 0 |
+| [swift](samples/swift.txt) | 467 | 467 | 0 |
+| [systemd](samples/systemd.txt) | 61 | 61 | 0 |
+| [tcl](samples/tcl.txt) | 71 | 71 | 0 |
+| [terraform](samples/terraform.txt) | 198 | 198 | 4 |
+| [tex](samples/tex.txt) | 177 | 177 | 0 |
+| [toml](samples/toml.txt) | 286 | 286 | 0 |
+| [twig](samples/twig.txt) | 259 | 259 | 12 |
+| [typescript](samples/typescript.txt) | 237 | 237 | 0 |
+| [v](samples/v.txt) | 90 | 89 | 1 |
+| [vala](samples/vala.txt) | 123 | 123 | 0 |
+| [verilog](samples/verilog.txt) | 269 | 269 | 0 |
+| [vhdl](samples/vhdl.txt) | 253 | 253 | 0 |
+| [vi](samples/vi.txt) | 80 | 80 | 0 |
+| [vue](samples/vue.txt) | 533 | 640 | 109 |
+| [xml](samples/xml.txt) | 110 | 110 | 0 |
+| [xresources](samples/xresources.txt) | 33 | 33 | 0 |
+| [yaml](samples/yaml.txt) | 106 | 106 | 0 |
+| [yum](samples/yum.txt) | 42 | 42 | 0 |
+| [zig](samples/zig.txt) | 153 | 153 | 0 |
+| [zscript](samples/zscript.txt) | 229 | 229 | 0 |
+| [zsh](samples/zsh.txt) | 125 | 125 | 0 |
+
+## Excluded YAML files
+
+
+
+## Differences
+
+### PowerShell.yaml
+
+- Line 23, column 2: micro `constant.string`, jmi `constant.specialChar`; "\"\\\"\""
+
+### ats.yaml
+
+- Line 32, column 11: micro `default`, jmi `constant.number`; "%{#sample 42%}"
+- Line 32, column 12: micro `default`, jmi `constant.number`; "%{#sample 42%}"
+
+### crystal.yaml
+
+- Line 14, column 11: micro `constant.number`, jmi `default`; "\"#{sample 42}\""
+- Line 14, column 12: micro `constant.number`, jmi `default`; "\"#{sample 42}\""
+
+### csx.yaml
+
+- Line 1, column 2: micro `symbol.operator`, jmi `preproc`; "#!"
+
+### d.yaml
+
+- Line 27, column 3: micro `constant.string`, jmi `constant.specialChar`; "r\"\\x\""
+- Line 27, column 4: micro `constant.string`, jmi `constant.specialChar`; "r\"\\x\""
+- Line 31, column 3: micro `constant.string`, jmi `constant.specialChar`; "x\"\\x\""
+
+### gleam.yaml
+
+- Line 17, column 3: micro `default`, jmi `identifier`; "#(sample 42)"
+- Line 17, column 4: micro `default`, jmi `identifier`; "#(sample 42)"
+- Line 17, column 5: micro `default`, jmi `identifier`; "#(sample 42)"
+
+### groff.yaml
+
+- Line 1, column 3: micro `statement`, jmi `type`; ".ds a]"
+
+### groovy.yaml
+
+- Line 22, column 13: micro `constant.number`, jmi `identifier`; "\"\"\"${sample 42}\"\"\""
+- Line 22, column 14: micro `constant.number`, jmi `identifier`; "\"\"\"${sample 42}\"\"\""
+
+### hare.yaml
+
+- Line 16, column 2: micro `constant.string`, jmi `error`; "'a'"
+
+### html.yaml
+
+- Line 11, column 3: micro `symbol.tag`, jmi `symbol`; "<a=>"
+- Line 15, column 17: micro `default`, jmi `constant.number`; "<scriptx>sample 42</scriptx>"
+- Line 15, column 18: micro `default`, jmi `constant.number`; "<scriptx>sample 42</scriptx>"
+
+### html4.yaml
+
+- Line 1, column 2: micro `default`, jmi `error`; "<ax>"
+- Line 1, column 3: micro `default`, jmi `error`; "<ax>"
+- Line 13, column 1: micro `symbol.tag`, jmi `comment`; "<!--x-->"
+
+### html5.yaml
+
+- Line 1, column 2: micro `default`, jmi `error`; "<ax>"
+- Line 1, column 3: micro `default`, jmi `error`; "<ax>"
+- Line 13, column 1: micro `symbol.tag`, jmi `comment`; "<!--x-->"
+
+### javascript.yaml
+
+- Line 33, column 3: micro `comment`, jmi `constant.specialChar`; "/*\\x*/"
+- Line 33, column 4: micro `comment`, jmi `constant.specialChar`; "/*\\x*/"
+
+### jinja2.yaml
+
+- Line 2, column 1: micro `default`, jmi `symbol.brackets`; "{%-sample 42-%}"
+- Line 2, column 2: micro `default`, jmi `symbol.operator`; "{%-sample 42-%}"
+- Line 2, column 3: micro `default`, jmi `symbol.operator`; "{%-sample 42-%}"
+
+### jsonnet.yaml
+
+- Line 15, column 3: micro `constant.string`, jmi `constant.specialChar`; "@\"\\\"\""
+- Line 15, column 4: micro `constant.string`, jmi `constant.specialChar`; "@\"\\\"\""
+- Line 17, column 3: micro `constant.string`, jmi `constant.specialChar`; "@'\\''"
+
+### lua.yaml
+
+- Line 34, column 6: micro `comment.block`, jmi `todo`; "--[=[TODO:]=]"
+- Line 34, column 7: micro `comment.block`, jmi `todo`; "--[=[TODO:]=]"
+- Line 34, column 8: micro `comment.block`, jmi `todo`; "--[=[TODO:]=]"
+
+### nftables.yaml
+
+- Line 18, column 8: micro `indent-char.whitespace`, jmi `comment`; "\"hello\" # comment"
+
+### objc.yaml
+
+- Line 15, column 1: micro `preproc`, jmi `special`; " # import \"x/\" "
+- Line 15, column 2: micro `preproc`, jmi `special`; " # import \"x/\" "
+- Line 15, column 3: micro `preproc`, jmi `special`; " # import \"x/\" "
+
+### octave.yaml
+
+- Line 14, column 2: micro `constant.string`, jmi `constant.specialChar`; "\"%\""
+- Line 19, column 2: micro `constant.string`, jmi `constant.specialChar`; "'%'"
+
+### pascal.yaml
+
+- Line 10, column 3: micro `constant.string`, jmi `constant.specialChar`; "#0\\x"
+- Line 10, column 4: micro `constant.string`, jmi `constant.specialChar`; "#0\\x"
+
+### php.yaml
+
+- Line 13, column 1: micro `default`, jmi `constant.string.url`; "ftps://a"
+- Line 13, column 2: micro `default`, jmi `constant.string.url`; "ftps://a"
+- Line 13, column 3: micro `default`, jmi `constant.string.url`; "ftps://a"
+
+### prql.yaml
+
+- Line 24, column 5: micro `constant.string`, jmi `constant.specialChar`; "f\"\"\"\\000\"\"\""
+- Line 24, column 6: micro `constant.string`, jmi `constant.specialChar`; "f\"\"\"\\000\"\"\""
+- Line 24, column 7: micro `constant.string`, jmi `constant.specialChar`; "f\"\"\"\\000\"\"\""
+
+### rust.yaml
+
+- Line 10, column 3: micro `constant.string`, jmi `constant.specialChar`; "b\"\\x\""
+- Line 10, column 4: micro `constant.string`, jmi `constant.specialChar`; "b\"\\x\""
+
+### stata.yaml
+
+- Line 2, column 10: micro `default`, jmi `identifier.macro`; "`\"`sample 42'\"'"
+- Line 2, column 11: micro `constant.number`, jmi `identifier.macro`; "`\"`sample 42'\"'"
+- Line 2, column 12: micro `constant.number`, jmi `identifier.macro`; "`\"`sample 42'\"'"
+
+### terraform.yaml
+
+- Line 14, column 3: micro `constant.string`, jmi `constant.specialChar`; "''%x''"
+- Line 14, column 4: micro `constant.string`, jmi `constant.specialChar`; "''%x''"
+- Line 15, column 3: micro `constant.string`, jmi `constant.specialChar`; "''\\a''"
+
+### twig.yaml
+
+- Line 2, column 4: micro `symbol.tag`, jmi `identifier`; "{{ abs }}"
+- Line 2, column 5: micro `symbol.tag`, jmi `identifier`; "{{ abs }}"
+- Line 2, column 6: micro `symbol.tag`, jmi `identifier`; "{{ abs }}"
+
+### v.yaml
+
+- Line 15, column 2: micro `symbol.attribute`, jmi `default`; "[x]"
+
+### vue.yaml
+
+- Line 2, column 13: micro `default`, jmi `error`; "<templatex><ax></templatex>"
+- Line 2, column 14: micro `default`, jmi `error`; "<templatex><ax></templatex>"
+- Line 4, column 13: micro `default`, jmi `symbol.tag.extended`; "<templatex></BODY x></templatex>"
+

package/apps/jsmdcui/tests/syntax-parity/compare.mjs

@@ -0,0 +1,75 @@
+import {resolve,dirname,join} from 'node:path';
+import {mkdir} from 'node:fs/promises';
+import {SyntaxDefinition,parseHeaderYaml} from '../../src/highlight/parser.js';
+import {Highlighter} from '../../src/highlight/highlighter.js';
+const here=dirname(Bun.fileURLToPath(import.meta.url));
+const local=resolve(here,'../../runtime/syntax');
+const upstream=resolve(process.argv[2]||'../go-micro');
+const oracle=resolve(process.argv[3]||'/tmp/jmi-micro-go/oracle');
+const generate=process.argv.includes('--generate');
+const upstreamSyntax=process.argv.includes('--upstream-syntax');
+const files=[],excluded=[],definitions=[],samples={};
+const definitionsByFile=new Map();
+for(const name of [...new Bun.Glob('*.yaml').scanSync(local)].sort()){
+ const text=await Bun.file(join(upstreamSyntax?join(upstream,'runtime/syntax'):local,name)).text();
+ const source=Bun.YAML.parse(text);const def=new SyntaxDefinition(parseHeaderYaml(source),source,text);definitions.push(def);definitionsByFile.set(name,def);
+ const orig=Bun.file(join(upstream,'runtime/syntax',name));
+ if(!await orig.exists()||await orig.text()!==text){excluded.push(name);continue;}
+ files.push(name);
+ if(!generate)samples[name]=await Bun.file(join(here,'samples',name.replace(/\.yaml$/,'.txt'))).text();
+}
+const child=Bun.spawn([oracle,join(upstream,'runtime/syntax')],{stdin:'pipe',stdout:'pipe',stderr:'inherit'});
+child.stdin.write(JSON.stringify({files,samples}));
+child.stdin.end();
+const raw=await new Response(child.stdout).text();
+const original=JSON.parse(raw);
+if(await child.exited)throw Error('micro oracle failed');
+await mkdir(join(here,'samples'),{recursive:true});
+const report={syntaxSource:upstreamSyntax?'upstream':'local',upstreamCommit:(await new Response(Bun.spawn(['git','-C',upstream,'rev-parse','HEAD'],{stdout:'pipe'}).stdout).text()).trim(),excluded,results:[]};
+const palette=new Map();
+for(const line of (await Bun.file(join(local,'../colorschemes/monokai.micro')).text()).split('\n')){
+ const m=line.match(/^color-link (\S+) "([^"]+)"/);if(m)palette.set(m[1],m[2]);
+}
+function color(group){
+ let value=palette.get(group);
+ while(!value&&group.includes('.')){group=group.slice(0,group.lastIndexOf('.'));value=palette.get(group);}
+ if(value==='default'||!value)value=palette.get('default');
+ return value.match(/#[0-9a-fA-F]{6}/)?.[0]||'#F8F8F2';
+}
+const esc=Bun.escapeHTML;
+function painted(text,groups){return text.split('\n').map((line,row)=>{
+ const chars=[...line];let out='',start=0;
+ while(start<chars.length){const g=groups[row]?.[start]||'default';let end=start+1;
+  while(end<chars.length&&(groups[row]?.[end]||'default')===g)end++;
+  out+=`<span title="${esc(g)}" style="color:${color(g)}">${esc(chars.slice(start,end).join(''))}</span>`;start=end;
+ }return out;
+}).join('\n');}
+let html='<!doctype html><meta charset="utf-8"><title>micro / jmi syntax comparison</title><style>body{background:#181818;color:#eee;font:16px sans-serif}section{margin:2em 0} .pair{display:grid;grid-template-columns:1fr 1fr;gap:1em}pre{background:#282828;padding:1em;overflow:auto;font:14px monospace}summary{cursor:pointer}a{color:#66d9ef}</style><h1>micro / jmi syntax comparison</h1><p>Same Monokai palette. Hover over a character to see its syntax group. Samples are lexical test fragments, not necessarily valid programs.</p>';
+for(const name of files){
+ const ref=original[name],def=definitionsByFile.get(name);
+ if(generate)await Bun.write(join(here,'samples',name.replace(/\.yaml$/,'.txt')),ref.text);
+ const actual=new Highlighter(def,definitions).highlightString(ref.text);
+ let coloredMicro=0,coloredJmi=0,different=0;const examples=[],actualGroups=[];
+ ref.text.split('\n').forEach((line,row)=>{
+  let offset=0,active='default';const chars=[...line],groups=[];actualGroups.push(groups);
+  for(let col=0;col<chars.length;col++){
+   if(actual[row].has(offset))active=actual[row].get(offset);
+   groups.push(active);
+   const expected=ref.groups[row]?.[col]||'default';
+   if(expected!=='default')coloredMicro++;if(active!=='default')coloredJmi++;
+   if(expected!==active){different++;if(examples.length<8)examples.push({line:row+1,column:col+1,text:line,micro:expected,jmi:active});}
+   offset+=chars[col].length;
+  }
+ });
+ html+=`<details><summary>${esc(name)} — ${different} different characters</summary><div class="pair"><div><h3>micro</h3><pre>${painted(ref.text,ref.groups)}</pre></div><div><h3>jmi</h3><pre>${painted(ref.text,actualGroups)}</pre></div></div></details>`;
+ report.results.push({file:name,filetype:ref.filetype,coloredMicro,coloredJmi,different,examples});
+}
+await Bun.write(join(here,'comparison.html'),html);
+await Bun.write(join(here,'report.json'),JSON.stringify(report,null,2)+'\n');
+const matched=report.results.filter(r=>!r.different),diff=report.results.filter(r=>r.different),uncolored=report.results.filter(r=>!r.coloredMicro||!r.coloredJmi);
+let md=`# micro / jmi syntax parity\n\nUpstream commit: ${report.upstreamCommit}\n\nSyntax source: ${report.syntaxSource}.\n\n${files.length} byte-identical YAML files tested; ${excluded.length} changed/missing files excluded.\n${matched.length} match; ${diff.length} differ; ${uncolored.length} lack non-default groups in one or both engines.\n\nComparison is by syntax group for every character, not theme RGB values.\nThese generated lexical samples exercise rule witnesses, not complete valid programs or exhaustive language coverage.\n\n| Syntax | micro colored chars | jmi colored chars | Different chars |\n|---|---:|---:|---:|\n`;
+for(const r of report.results)md+=`| [${r.filetype}](samples/${r.file.replace(/\.yaml$/,'.txt')}) | ${r.coloredMicro} | ${r.coloredJmi} | ${r.different} |\n`;
+md+='\n## Excluded YAML files\n\n'+excluded.map(x=>'- '+x).join('\n')+'\n\n## Differences\n\n';
+for(const r of diff){md+=`### ${r.file}\n\n`;for(const e of r.examples.slice(0,3))md+=`- Line ${e.line}, column ${e.column}: micro \`${e.micro}\`, jmi \`${e.jmi}\`; ${JSON.stringify(e.text)}\n`;md+='\n';}
+await Bun.write(join(here,'REPORT.md'),md);
+console.log(JSON.stringify({files:files.length,excluded,matched:matched.length,differ:diff.length,uncolored:uncolored.map(r=>r.file)},null,2));

package/apps/jsmdcui/tests/syntax-parity/go.mod

@@ -0,0 +1,5 @@
+module syntax-parity
+
+go 1.24
+
+require github.com/micro-editor/micro/v2 v2.0.0