tree-sitter-ucode 0.3.0 → 0.4.0

package/README.md

@@ -6,13 +6,13 @@ Two grammars are provided:
 
 | Grammar | Scope | File types |
 |---------|-------|------------|
-| `ucode` | `source.uc` | `.uc` |
-| `ucode_tmpl` | `source.uc.tmpl` | `.uc` (template files — detected by content) |
+| `ucode` | `source.uc` | `.uc`, `.ucode`, `.ut` |
+| `ucode_markup` | `source.ucode.markup` | `.uc`, `.ucode`, `.ut`, `.uc.tmpl` (template files — detected by content) |
 
-Both grammars use the `.uc` extension. Template files are distinguished from plain
-code files by content: any `.uc` file whose first tag opener (`{%`, `{{`, or `{#`)
-appears at the start of a line is automatically parsed by `ucode_tmpl`. Plain code
-files fall back to `ucode`. See [File-type detection](#file-type-detection) below.
+Both grammars share the same file extensions. Template files are distinguished from plain
+code files by content: any file whose first tag opener (`{%`, `{{`, or `{#`) appears at
+the start of a line is automatically parsed by `ucode_markup`. Plain code files fall back
+to `ucode`. See [File-type detection](#file-type-detection) below.
 
 ## Ucode vs JavaScript
 
@@ -47,79 +47,37 @@ To regenerate parsers after editing a grammar file:
 # ucode grammar
 npx tree-sitter generate
 
-# ucode_tmpl grammar
-npx tree-sitter generate tmpl/grammar.js --output tmpl/src
+# ucode_markup grammar (generated from grammar.js — do not edit markup/grammar.js directly)
+node scripts/generate-markup-grammar.js
+cd markup && npx tree-sitter generate
 ```
 
 ## Test
 
 ```sh
-npm test             # runs tree-sitter test for ucode and ucode_tmpl
+npm test             # runs tree-sitter test for ucode and ucode_markup
 ```
 
 To filter by corpus file name:
 
 ```sh
-npx tree-sitter test --lib-path ucode.so --lang-name ucode --file-name control_flow
-npx tree-sitter test -p tmpl --file-name template
+npx tree-sitter test --file-name control_flow
+cd markup && npx tree-sitter test --file-name markup
 ```
 
 ## File-type detection
 
-Both grammars claim the `.uc` extension. Tools that respect `content-regex` in
+Both grammars claim the same file extensions. Tools that respect `content-regex` in
 `tree-sitter.json` (including the tree-sitter CLI ≥ 0.24) automatically route
-template files to `ucode_tmpl` when a tag opener appears at the start of a line.
+template files to `ucode_markup` when a tag opener appears at the start of a line.
 Editors that manage their own filetype dispatch (Neovim, Helix) need an explicit
 rule — see the editor sections below.
 
-## Use in Neovim (nvim-treesitter)
-
-Add to your nvim-treesitter config (e.g. `~/.config/nvim/lua/plugins/treesitter.lua`):
-
-```lua
-local parser_config = require("nvim-treesitter.parsers").get_parser_configs()
-
-parser_config.ucode = {
-  install_info = {
-    url = "https://github.com/m00qek/tree-sitter-ucode",
-    files = { "src/parser.c", "src/scanner.c" },
-    branch = "main",
-  },
-  filetype = "ucode",
-}
-
-parser_config.ucode_tmpl = {
-  install_info = {
-    url = "https://github.com/m00qek/tree-sitter-ucode",
-    files = { "tmpl/src/parser.c", "tmpl/src/scanner.c" },
-    branch = "main",
-  },
-  filetype = "ucode_tmpl",
-}
-```
+## Use in Neovim
 
-Associate `.uc` files with the right filetype. Since nvim-treesitter does not
-use `content-regex`, the mapping must be done with a `BufRead` autocmd or a
-filetype function that inspects the file content:
-
-```lua
-vim.filetype.add({
-  extension = {
-    uc = function(path)
-      -- Files whose first tag opener is at the start of a line are templates
-      local f = io.open(path, "r")
-      if f then
-        local content = f:read("*a")
-        f:close()
-        if content:find("^%s*{[%%{#]", 1, false) or content:find("\n%s*{[%%{#]") then
-          return "ucode_tmpl"
-        end
-      end
-      return "ucode"
-    end,
-  },
-})
-```
+The easiest way to install this grammar in Neovim is with
+[tree-sitter-manager.nvim](https://github.com/m00qek/tree-sitter-manager.nvim),
+which handles parser registration, filetype detection, and query setup automatically.
 
 ## Use in Helix
 
@@ -129,37 +87,37 @@ Add to `~/.config/helix/languages.toml`:
 [[language]]
 name          = "ucode"
 scope         = "source.uc"
-file-types    = [{ glob = "*.uc" }]
+file-types    = [{ glob = "*.uc" }, { glob = "*.ucode" }, { glob = "*.ut" }]
 comment-token = "//"
 indent        = { tab-width = 2, unit = "  " }
 grammar       = "ucode"
 
 [[language]]
-name          = "ucode-tmpl"
-scope         = "source.uc.tmpl"
+name          = "ucode-markup"
+scope         = "source.ucode.markup"
+file-types    = [{ glob = "*.uc.tmpl" }]
 comment-token = "{#"
 indent        = { tab-width = 2, unit = "  " }
-grammar       = "ucode_tmpl"
+grammar       = "ucode_markup"
 
 [[grammar]]
 name   = "ucode"
-source = { git = "https://github.com/m00qek/tree-sitter-ucode", rev = "v0.3.0" }
+source = { git = "https://github.com/m00qek/tree-sitter-ucode", rev = "v0.4.0" }
 
 [[grammar]]
-name   = "ucode_tmpl"
-source = { git = "https://github.com/m00qek/tree-sitter-ucode", rev = "v0.3.0", subpath = "tmpl" }
+name   = "ucode_markup"
+source = { git = "https://github.com/m00qek/tree-sitter-ucode", rev = "v0.4.0", subpath = "markup" }
 ```
 
-Helix does not support content-based filetype detection. To open a `.uc`
-template file as `ucode-tmpl`, use `:set-language ucode-tmpl` in command mode,
-or configure a file-specific override via a `.helix/languages.toml` in your
-project.
+Helix does not support content-based filetype detection for shared extensions. For
+`.uc` files that are templates, use `:set-language ucode-markup` in command mode,
+or configure a file-specific override via a `.helix/languages.toml` in your project.
 
 ## Template files
 
-Template files mix raw text with code tags. The `ucode_tmpl` grammar produces a
-`document` tree; editors use language injection to apply ucode highlighting
-inside the code and expression tags.
+Template files mix raw text with code tags. The `ucode_markup` grammar produces a
+`markup` tree; editors use language injection to apply ucode highlighting inside the
+code and expression tags.
 
 | Tag | Purpose |
 |-----|---------|
@@ -176,13 +134,6 @@ with any closer variant. `{%-` / `{{-` / `{#-` strip the preceding raw text;
 `-%}` / `-}}` / `-#}` strip the following raw text. `{%+` suppresses
 `lstrip_blocks` stripping and may be combined with `-%}`.
 
-**EOF-implicit close**: a `{%` statement tag that reaches end-of-file without
-an explicit `%}` is treated as implicitly closed. This supports the common
-OpenWrt pattern of a file that opens one `{%` block at the top and contains
-only ucode code, with no closing `%}`. The parse tree shows an `(eof_close)`
-node in the `close` field. Expression (`{{`) and comment (`{#`) tags still
-require explicit closers.
-
 Example:
 
 ```

package/grammar.js

@@ -14,9 +14,19 @@ module.exports = grammar({
   name: 'ucode',
 
   externals: $ => [
-    $._automatic_semicolon,
-    $._template_chars,
-    $._ternary_qmark,
+    $._automatic_semicolon,        //  0
+    $._template_chars,             //  1
+    $._ternary_qmark,              //  2
+    $.raw_text,                    //  3  literal text outside tags
+    $.statement_tag_open,          //  4  {%
+    $.statement_tag_trim_open,     //  5  {%-
+    $.statement_tag_lstrip_open,   //  6  {%+
+    $.statement_tag_close,         //  7  %}
+    $.statement_tag_trim_close,    //  8  -%}
+    $.expression_tag_open,         //  9  {{
+    $.expression_tag_trim_open,    // 10  {{-
+    $.expression_tag_close,        // 11  }}
+    $.expression_tag_trim_close,   // 12  -}}
   ],
 
   extras: $ => [
@@ -73,6 +83,12 @@ module.exports = grammar({
     $._identifier,
     $._reserved_identifier,
     $._lhs_expression,
+    $._markup_node,
+    $._if_markup_node,
+    $._stmt_open,
+    $._stmt_close,
+    $._expr_open,
+    $._expr_close,
   ],
 
   precedences: $ => [
@@ -120,6 +136,63 @@ module.exports = grammar({
       repeat($.statement),
     ),
 
+    //
+    // Markup entry point
+    //
+    // A .uc.tmpl document is a flat sequence of markup nodes: raw
+    // text, comment tags, expression tags, statement tags, and the
+    // alt-syntax constructs that span multiple tags.
+    //
+    // Statement tags that contain only simple (non-spanning) code are
+    // wrapped in `statement_tag`.  Alt-syntax constructs that span tag
+    // boundaries appear directly as markup nodes with explicit tag-open /
+    // tag-close fields, giving a pristine tree with no empty-statement noise.
+    //
+    markup: $ => seq(
+      optional($.hash_bang_line),
+      repeat($._markup_node),
+    ),
+
+    _markup_node: $ => choice(
+      $.raw_text,
+      $.expression_tag,
+      $.comment_tag,
+      $.statement_tag,
+      // Alt-syntax constructs that span tag boundaries:
+      $.if_alt_statement,
+      $.for_alt_statement,
+      $.for_in_alt_statement,
+      $.while_alt_statement,
+    ),
+
+    // -----------------------------------------------------------------------
+    // Simple tag wrappers
+    // -----------------------------------------------------------------------
+
+    // A statement_tag wraps non-spanning code: {% stmt; stmt; %}
+    statement_tag: $ => seq(
+      field('open',  $._stmt_open),
+      repeat($.statement),
+      field('close', $._stmt_close),
+    ),
+
+    // {{ expr }} or {{- expr -}}
+    expression_tag: $ => seq(
+      field('open',  $._expr_open),
+      optional($._expressions),
+      field('close', $._expr_close),
+    ),
+
+    // {# ... #}  with optional whitespace-stripping markers
+    comment_tag: $ => seq(
+      field('open',    choice('{#-', '{#')),
+      optional(field('content', $.comment_content)),
+      field('close',   choice('-#}', '#}')),
+    ),
+
+    // Matches everything up to but not including #} or -#}
+    comment_content: _ => /([^#-]|#[^}]|-(?:[^#]|#[^}]))+/,
+
     hash_bang_line: _ => /#!.*/,
 
     //
@@ -285,22 +358,71 @@ module.exports = grammar({
       optional(field('alternative', $.else_clause)),
     )),
 
-    // Alternative colon/endif syntax
-    if_alt_statement: $ => seq(
-      'if',
+    // Alternative colon/endif syntax — two forms:
+    //   code form:   if (cond): stmts … endif        (used in program / statement_tag)
+    //   markup form: {% if (cond): %} … {% endif %}   (spans tag boundaries in markup)
+    //
+    // The markup form uses a flat content repeat (_if_markup_node) rather than
+    // nested elif/else bodies.  elif_clause_tag and else_alt_clause_tag are pure
+    // header tags that appear as regular items inside that repeat.  This avoids
+    // the shift/reduce conflict that arises when a nested repeat($._markup_node)
+    // can't decide whether statement_tag_open starts another body node or the
+    // enclosing end tag.
+    if_alt_statement: $ => choice(
+      seq(
+        'if',
+        field('condition', $.parenthesized_expression),
+        ':',
+        field('body', repeat($.statement)),
+        repeat(field('elif_clause', $.elif_clause)),
+        optional(field('else_body', $.else_alt_clause)),
+        'endif',
+      ),
+      seq(
+        field('open',    $._stmt_open),
+        'if',
+        field('condition', $.parenthesized_expression),
+        ':',
+        repeat($.statement),
+        field('close',   $._stmt_close),
+        repeat($._if_markup_node),
+        field('end_open',  $._stmt_open),
+        'endif',
+        field('end_close', $._stmt_close),
+      ),
+    ),
+
+    // Flat content node for if_alt_statement markup bodies.
+    // elif_clause_tag and else_alt_clause_tag are plain header tags here;
+    // the actual body content between them is expressed as sibling nodes.
+    _if_markup_node: $ => choice(
+      $._markup_node,
+      $.elif_clause_tag,
+      $.else_alt_clause_tag,
+    ),
+
+    // Inline wrappers for tag delimiter tokens.
+    // Each groups all variants (plain / trim / lstrip) so grammar rules stay
+    // concise while still surfacing distinct node types for highlight queries.
+    _stmt_open:  $ => choice($.statement_tag_open, $.statement_tag_trim_open, $.statement_tag_lstrip_open),
+    _stmt_close: $ => choice($.statement_tag_close, $.statement_tag_trim_close),
+    _expr_open:  $ => choice($.expression_tag_open, $.expression_tag_trim_open),
+    _expr_close: $ => choice($.expression_tag_close, $.expression_tag_trim_close),
+
+    elif_clause: $ => seq(
+      'elif',
       field('condition', $.parenthesized_expression),
       ':',
       field('body', repeat($.statement)),
-      repeat(field('elif_clause', $.elif_clause)),
-      optional(field('else_body', $.else_alt_clause)),
-      'endif',
     ),
 
-    elif_clause: $ => seq(
+    // Markup form of elif: just the header tag; body is sibling _if_markup_nodes
+    elif_clause_tag: $ => seq(
+      field('open',      $._stmt_open),
       'elif',
       field('condition', $.parenthesized_expression),
       ':',
-      field('body', repeat($.statement)),
+      field('close',     $._stmt_close),
     ),
 
     else_alt_clause: $ => seq(
@@ -308,6 +430,13 @@ module.exports = grammar({
       field('body', repeat($.statement)),
     ),
 
+    // Markup form of else: just the header tag; body is sibling _if_markup_nodes
+    else_alt_clause_tag: $ => seq(
+      field('open',  $._stmt_open),
+      'else',
+      field('close', $._stmt_close),
+    ),
+
     switch_statement: $ => seq(
       'switch',
       field('value', $.parenthesized_expression),
@@ -319,11 +448,24 @@ module.exports = grammar({
       field('body', $.statement),
     ),
 
-    for_alt_statement: $ => seq(
-      forHeader($),
-      ':',
-      field('body', repeat($.statement)),
-      'endfor',
+    for_alt_statement: $ => choice(
+      seq(
+        forHeader($),
+        ':',
+        field('body', repeat($.statement)),
+        'endfor',
+      ),
+      seq(
+        field('open',    $._stmt_open),
+        forHeader($),
+        ':',
+        repeat($.statement),
+        field('close',   $._stmt_close),
+        field('body',    repeat($._markup_node)),
+        field('end_open',  $._stmt_open),
+        'endfor',
+        field('end_close', $._stmt_close),
+      ),
     ),
 
     for_in_statement: $ => seq(
@@ -332,12 +474,42 @@ module.exports = grammar({
       field('body', $.statement),
     ),
 
-    for_in_alt_statement: $ => seq(
-      'for',
-      $._for_header,
-      ':',
-      field('body', repeat($.statement)),
-      'endfor',
+    for_in_alt_statement: $ => choice(
+      seq(
+        'for',
+        $._for_header,
+        ':',
+        field('body', repeat($.statement)),
+        'endfor',
+      ),
+      seq(
+        field('open',    $._stmt_open),
+        'for',
+        $._for_header,
+        ':',
+        repeat($.statement),
+        field('close',   $._stmt_close),
+        field('body',    repeat($._markup_node)),
+        field('end_open',  $._stmt_open),
+        'endfor',
+        field('end_close', $._stmt_close),
+      ),
+      // Compact double-nested form: {% for (outer): for (inner): %} body {% endfor; endfor %}
+      // Both iterables contribute `right` fields; both loop vars contribute `left` fields.
+      seq(
+        field('open',    $._stmt_open),
+        'for',
+        $._for_header,
+        ':',
+        'for',
+        $._for_header,
+        ':',
+        field('close',   $._stmt_close),
+        field('body',    repeat($._markup_node)),
+        field('end_open',  $._stmt_open),
+        'endfor', ';', 'endfor',
+        field('end_close', $._stmt_close),
+      ),
     ),
 
     // Supports both `for (k in obj)` and `for (k, v in obj)` (ucode two-variable form)
@@ -365,12 +537,26 @@ module.exports = grammar({
       field('body', $.statement),
     ),
 
-    while_alt_statement: $ => seq(
-      'while',
-      field('condition', $.parenthesized_expression),
-      ':',
-      field('body', repeat($.statement)),
-      'endwhile',
+    while_alt_statement: $ => choice(
+      seq(
+        'while',
+        field('condition', $.parenthesized_expression),
+        ':',
+        field('body', repeat($.statement)),
+        'endwhile',
+      ),
+      seq(
+        field('open',    $._stmt_open),
+        'while',
+        field('condition', $.parenthesized_expression),
+        ':',
+        repeat($.statement),
+        field('close',   $._stmt_close),
+        field('body',    repeat($._markup_node)),
+        field('end_open',  $._stmt_open),
+        'endwhile',
+        field('end_close', $._stmt_close),
+      ),
     ),
 
     try_statement: $ => seq(