rapydscript-ng 0.7.24 → 0.8.0

package/CHANGELOG.md

@@ -1,3 +1,22 @@
+version 0.8.0
+=======================
+
+  * Implement generation of source maps via the --source-map option
+  * Implement dead code removal via the --tree-shaking option
+  * Add support for async and await
+  * Implement a ``fmt`` command for code formatting and an ``lsp`` command for a Language Server
+  * Implement a tree sitter grammar so that robust syntax highlighting is potentially available in all editors
+  * Implement a neovim plugin that enables treesitter based syntax highlighting and LSP server for ``.pyj`` files
+  * Implement support for importing user modules in the embedded compiler
+  * Double the compilation speed, especially useful for large projects
+  * Drop support for ES5
+  * List objects are now unmodified JS Arrays for greater performance. Python
+    list methods continue to work.
+  * Change the cache format to cache only the AST and not final output so that
+    the cache can be re-used regardless of output and tree shaking options
+  * Make the compiler internally use async IO. This means the embedded compiler
+    functions are also async.
+ 
 version 0.7.24
 =======================
 

package/README.md

@@ -42,11 +42,13 @@ backwards compatible) features. For more on the forking, [see the bottom of this
   - [Method Binding](#method-binding)
 - [Iterators](#iterators)
 - [Generators](#generators)
+- [Asnyc and Await](#async-and-await)
 - [Modules](#modules)
 - [Exception Handling](#exception-handling)
 - [Scope Control](#scope-control)
 - [Available Libraries](#available-libraries)
 - [Linter](#linter)
+- [Language Server (LSP)](#language-server-lsp)
 - [Making RapydScript even more pythonic](#making-rapydscript-even-more-pythonic)
 - [Advanced Usage Topics](#advanced-usage-topics)
     - [Browser Compatibility](#browser-compatibility)
@@ -1254,9 +1256,12 @@ syntax.
 Generators create JavaScript iterator objects. For differences between python
 and JavaScript iterators, see the section on iterators above. 
 
-Currently, generators are down-converted to ES 5 switch statements. In the
-future, when ES 6 support is widespread, they will be converted to native
-JavaScript ES 6 generators.
+Async and Await
+------------------
+
+Python's ``async`` and ``await`` keywords are supported, but note that async
+functions return JavaScript Promise objects not co-routines. This is so that
+they interoperate with the JS ecosystem and browser APIs in particular.
 
 Modules
 -------
@@ -1427,6 +1432,41 @@ that the linter will not raise undefined errors for. You can turn off
 individual checks that you do not find useful. See ``rapydscript lint -h`` for
 details.
 
+Language Server (LSP)
+-----------------------
+
+RapydScript ships with a Language Server Protocol server so that editors can
+offer rich language support for ``.pyj`` files. Start it with:
+
+	rapydscript lsp
+
+The server talks the standard LSP JSON-RPC protocol over stdin/stdout, so it is
+meant to be launched by an editor/LSP client rather than run interactively. It
+provides:
+
+- **Code completion** — in-scope symbols, builtins and keywords, plus member
+  completion for imported modules (``mod.<complete>``).
+- **Diagnostics** — the same checks as the ``lint`` sub-command, plus warnings
+  for imports that cannot be resolved in the import path. The parser recovers
+  from syntax errors, so a file that is being edited is still analysed (and
+  multiple errors can be reported at once).
+- **Hover** — the kind, origin and docstring of the symbol under the cursor.
+- **Code actions** — quick fixes such as removing an unused import/local,
+  silencing a check with a ``# noqa`` comment, and formatting the document.
+- **Document formatting** — the same formatter as the ``fmt`` sub-command,
+  controlled by ``--line-length`` and ``--preferred-quote``.
+- **Go to definition** — including jumping into imported modules.
+- **Find all references** — resolved across files through the import graph.
+- **Rename** — renames a symbol everywhere it is used across the workspace.
+
+Imports are resolved with the same ``--import-path`` (``-p``) option as the
+``compile`` sub-command, so point it at your project's import directories. For
+example:
+
+	rapydscript lsp --import-path src:vendor --line-length 100
+
+See ``rapydscript lsp -h`` for the full list of options.
+
 Making RapydScript even more pythonic
 ---------------------------------------
 
@@ -1454,11 +1494,9 @@ Advanced Usage Topics
 
 #### Browser Compatibility
 
-RapydScript compiles your code such that it will work on browsers that are
-compatible with the ES 5 JavaScript standard. The compiler has a 
-``--js-version`` option that can also be used to output ES 6 only code. This
-code is smaller and faster than the ES 5 version, but is not as widely
-compatible.
+RapydScript compiles your code to ES 6 compatible JavaScript. Any modern
+browser or Node.js version that supports ES 6 will run the output. The
+``--js-version`` option only accepts ``6``.
 
 #### Tabs vs Spaces
 
@@ -1491,9 +1529,9 @@ To do so, simply include the [embeddable rapydscript compiler](https://sw.kovidg
 in your page, and use it to compile arbitrary RapydScript code. 
 
 You create the compiler by calling: `RapydScript.create_embedded_compiler()` and compile
-code with `compiler.compile(code)`. You can execute the resulting JavaScript
-using the standard `eval()` function. See the sample
-HTML below for an example.
+code with `await compiler.compile(code)`. Both calls return Promises, so you must use
+`async`/`await` (or `.then()`) to get the results. You can execute the resulting JavaScript
+using the standard `eval()` function. See the sample HTML below for an example.
 
 ```html
 <!DOCTYPE html>
@@ -1503,9 +1541,9 @@ HTML below for an example.
         <title>Test embedded RapydScript</title>
         <script charset="UTF-8" src="https://sw.kovidgoyal.net/rapydscript/repl/rapydscript.js"></script>
         <script>
-var compiler = RapydScript.create_embedded_compiler();
-var js = compiler.compile("def hello_world():\n a='RapydScript is cool!'\n print(a)\n alert(a)");
-window.onload = function() {
+window.onload = async function() {
+    var compiler = await RapydScript.create_embedded_compiler();
+    var js = await compiler.compile("def hello_world():\n a='RapydScript is cool!'\n print(a)\n alert(a)");
     document.body.textContent = js;
     eval(js);
     eval('hello_world()');
@@ -1519,20 +1557,28 @@ window.onload = function() {
 There are a couple of caveats when using the embedded compiler:
 
 * It only works when run in a modern browser (one that supports ES6) so no
-  Internet Explorer. You can have it work in an ES 5 runtime by passing
-  an option to the compile() method, like this:
-  ```
-  compiler.compile(code, {js_version:5})
-  ```
-  Note that doing this means that you cannot use generators and the
-  yield keyword in your RapydScript code.
+  Internet Explorer.
 
-* Importing of modules only works with the standard library modules. There is
-  currently no way to make your own modules importable.
+* To implement importing of modules you need to create a virtual file system
+  like this, before calling ``create_embedded_compiler()``:
 
-* To generate the embedded compiler yourself (rapydscript.js) from a source
-  checkout of rapydscript, follow the instructions above for installing from
-  source, then run `bin/web-repl-export /path/to/export/directory`
+  ```
+  RapydScript.virtual_file_system = {
+    read_file: my_read_file,
+    write_file: my_write_file,
+    stat_file: my_stat_file
+  }
+  ```
+  These functions should have the same signature and semantics as the 
+  ``fs.promises.readFile, fs.promises.writeFile and fs.promises.statFile``
+  functions from Node. Currently only the ``mtimeMs`` field from the stat
+  result is needed. They will be passed file paths starting with ``__vfs__/``
+  and should read/write/stat them as needed.
+
+* To generate the embedded compiler yourself (rapydscript.js) 
+  run `rapydscript web-repl-export /path/to/export/directory`.
+  Then you can simply open the ``index.html`` file in that directory in a browser
+  to see the web REPL in action.
 
 Internationalization
 -------------------------

package/bin/package.json

@@ -0,0 +1 @@
+{"type": "module"}

package/bin/rapydscript

@@ -1,70 +1,73 @@
-#!/usr/bin/env node 
+#!/usr/bin/env node
 // vim:ft=javascript:ts=4:et
 
-"use strict";
+import { fileURLToPath } from 'url';
+import path from 'path';
+import { spawn } from 'child_process';
+import * as utils from '../tools/utils.mjs';
+import { create_compiler } from '../tools/compiler.mjs';
 
-function load(mod) {
-    return require('../tools/' + mod);
-}
+const __filename = fileURLToPath(import.meta.url);
+
+var start_time = new Date().getTime();
+
+var { argv } = await import('../tools/cli.mjs');
+
+var base_path = path.normalize(path.join(path.dirname(__filename), ".."));
+var src_path = path.join(base_path, 'src');
+var lib_path = path.join(base_path, 'dev');
+if (!(await utils.path_exists(path.join(lib_path, 'compiler.js')))) lib_path = path.join(base_path, 'release');
 
-var utils = load('utils');
+// Create compiler instance once, then expose it synchronously via the global
+// so bundled modules (repl, gettext, etc.) can call create_rapydscript_compiler() synchronously.
+const _compiler_instance = await create_compiler();
+globalThis.create_rapydscript_compiler = () => _compiler_instance;
 
-// We need ES 6 generators so relaunch with the --harmony flag
-if (!utils.generators_available()) {
-    if (process.execArgv.indexOf('--harmony') != -1) {
-        console.error('RapydScript needs ES 6 generators, update your version of nodejs');
-        process.exit(1);
+if (argv.mode === 'self') {
+    if (argv.files.length > 0) {
+        console.error("WARN: Ignoring input files since --self was passed");
     }
-    var args = ['--harmony', module.filename].concat(process.argv.slice(2));
-    require('child_process').spawn(process.execPath, args, {stdio:'inherit'}).on('exit', function(code, signal) {
-        process.exit(code);
-    });
-} else {
-
-    var start_time = new Date().getTime();
-    var path = require('path');
-
-    var argv = load('cli').argv;
-
-    var base_path = path.normalize(path.join(path.dirname(module.filename), ".."));
-    var src_path = path.join(base_path, 'src');
-    var lib_path = path.join(base_path, 'dev');
-    if (!utils.path_exists(path.join(lib_path, 'compiler.js'))) lib_path = path.join(base_path, 'release');
-
-    if (argv.mode === 'self') {
-        if (argv.files.length > 0) {
-            console.error("WARN: Ignoring input files since --self was passed");
-        }
-        load('self')(base_path, src_path, lib_path, argv.complete, argv.profile);
-        if (argv.test) {
-            console.log('\nRunning test suite...\n');
-            argv.files = [];  // Ensure all tests are run
-            load('test')(argv, base_path, src_path, path.join(base_path, 'dev'));
-        }
-        process.exit(0);
-    } else 
-
-    if (argv.mode === 'test') {
-        load('test')(argv, base_path, src_path, lib_path);
-    } else 
-
-    if (argv.mode === 'lint') {
-        load('lint').cli(argv, base_path, src_path, lib_path);
-    } else
-
-    if (argv.mode === 'repl') {
-        load('repl')({'lib_path':lib_path, 'imp_path':path.join(src_path, 'lib'), show_js:!argv.no_js});
-    } else 
-    
-    if (argv.mode === 'gettext') {
-        load('gettext').cli(argv, base_path, src_path, lib_path);
-    } else
-    
-    if (argv.mode === 'msgfmt') {
-        load('msgfmt').cli(argv, base_path, src_path, lib_path);
-    } else
-    
-    {
-        load('compile')(start_time, argv, base_path, src_path, lib_path);
+    await (await import('../tools/self.mjs')).default(base_path, src_path, lib_path, argv.complete, argv.profile);
+    if (argv.test) {
+        console.log('\nRunning test suite...\n');
+        argv.files = [];  // Ensure all tests are run
+        await (await import('../tools/test.mjs')).default(argv, base_path, src_path, path.join(base_path, 'dev'));
     }
+    process.exit(0);
+} else
+
+if (argv.mode === 'test') {
+    await (await import('../tools/test.mjs')).default(argv, base_path, src_path, lib_path);
+} else
+
+if (argv.mode === 'lint') {
+    await (await import('../tools/lint.mjs')).cli(argv, base_path, src_path, lib_path);
+} else
+
+if (argv.mode === 'fmt') {
+    await (await import('../tools/fmt.mjs')).cli(argv, base_path, src_path, lib_path);
+} else
+
+if (argv.mode === 'lsp') {
+    await (await import('../tools/lsp.mjs')).cli(argv, base_path, src_path, lib_path);
+} else
+
+if (argv.mode === 'repl') {
+    await (await import('../tools/repl.mjs')).default({'lib_path':lib_path, 'imp_path':path.join(src_path, 'lib'), show_js:!argv.no_js});
+} else
+
+if (argv.mode === 'gettext') {
+    await (await import('../tools/gettext.mjs')).cli(argv, base_path, src_path, lib_path);
+} else
+
+if (argv.mode === 'msgfmt') {
+    (await import('../tools/msgfmt.mjs')).cli(argv, base_path, src_path, lib_path);
+} else
+
+if (argv.mode === 'web-repl-export') {
+    await (await import('../tools/web_repl_export.mjs')).default(base_path, lib_path, argv);
+} else
+
+{
+    await (await import('../tools/compile.mjs')).default(start_time, argv, base_path, src_path, lib_path);
 }

package/editor-plugins/nvim/rapydscript/README.md

@@ -0,0 +1,184 @@
+# rapydscript.nvim
+
+Neovim plugin that wires up the RapydScript LSP server (`rapydscript lsp`) as a
+native Neovim LSP client for `.pyj` files.
+
+## Features
+
+Provided by the LSP server — no extra plugins required:
+
+- **Completions** — in-scope symbols, builtins, keywords, and member completion (`mod.<TAB>`)
+- **Diagnostics** — lint checks and unresolved import warnings, updated as you type
+- **Hover** (`K`) — kind, origin, and docstring of the symbol under the cursor
+- **Go to definition** (`gd`) — including jumping into imported modules
+- **Find references** (`gr`) — resolved across files through the import graph
+- **Rename** (`<leader>rn`) — renames everywhere across the workspace
+- **Code actions** (`<leader>ca`) — remove unused import/local, add `# noqa`, format document
+- **Document formatting** (`<leader>f`) — same as `rapydscript fmt`
+- **Document symbols** — outline view via any symbols picker
+
+## Requirements
+
+- Neovim ≥ 0.10
+- `rapydscript` on your `$PATH` (install with `npm install -g rapydscript-ng`), or the
+  plugin directory must be inside a RapydScript repository checkout (the repo's own
+  `bin/rapydscript` is used automatically as a fallback)
+- **Syntax highlighting only**: a C compiler on your `$PATH` is required to compile the
+  tree-sitter parser the first time the plugin loads.  On Linux/macOS any `cc`-compatible
+  compiler works (gcc, clang, etc.).  On Windows, `cl` (MSVC) or `clang-cl` must be
+  available — run Neovim from a Visual Studio Developer Command Prompt, or add the
+  compiler to your `PATH` manually.  LSP features work regardless of whether compilation
+  succeeds.
+
+## Installation
+
+### lazy.nvim
+
+```lua
+{
+    dir = '/path/to/editor-plugins/nvim/rapydscript',
+    ft = 'rapydscript',
+    opts = {},
+}
+```
+
+**Example with options:**
+
+```lua
+{
+    dir = "/path/to/editor-plugins/nvim/rapydscript",
+    ft = "rapydscript",
+    opts = {
+        line_length   = 100,
+        preferred_quote = "double",
+    },
+}
+```
+
+### Manual (no plugin manager)
+
+Add the plugin directory to your runtime path and call `setup()`:
+
+```lua
+-- in ~/.config/nvim/init.lua
+vim.opt.rtp:prepend("/path/to/rapydscript/nvim-lsp-plugin")
+require("rapydscript").setup()
+```
+
+## Configuration reference
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `cmd` | `string[]` | `{"rapydscript","lsp"}` | Command used to start the server |
+| `import_path_patterns` | `string[]` | `{".", "src", "src/pyj"}` | Glob patterns relative to the project root; matching directories that contain `.pyj` files are passed automatically as `--import-path` |
+| `line_length` | `number\|nil` | `nil` | Max line length for the formatter, passed as `--line-length` |
+| `preferred_quote` | `string\|nil` | `nil` | `"single"` or `"double"`, passed as `--preferred-quote` |
+| `filetypes` | `string[]` | `{"rapydscript"}` | Filetypes that trigger server attachment |
+| `root_markers` | `string[]` | `{".git","package.json","rapydscript.json"}` | Files/dirs used to detect the project root |
+
+### Automatic import path detection
+
+When a project root is found (via `root_markers`), the plugin expands each pattern in
+`import_path_patterns` as a glob relative to that root.  Any matching directory that
+contains at least one `.pyj` file is added to the server's import search path
+automatically — no manual configuration needed for standard project layouts.
+
+If no project root is detected the setting has no effect.
+
+To add custom search locations alongside the defaults:
+
+```lua
+opts = {
+    import_path_patterns = { ".", "src", "src/pyj", "vendor/pyj" },
+}
+```
+
+To disable auto-detection entirely, pass an empty list:
+
+```lua
+opts = {
+    import_path_patterns = {},
+}
+```
+
+## Changing settings at runtime
+
+The server accepts live setting updates via the LSP
+`workspace/didChangeConfiguration` notification — no restart needed.  Call
+`require("rapydscript").update_settings(opts)` with any subset of the two
+settings keys:
+
+```lua
+require("rapydscript").update_settings({
+    line_length    = 100,               -- new max line length for the formatter
+    preferred_quote = "double",         -- "single" or "double"
+})
+```
+
+Only the keys you pass are updated; the rest keep their current values.
+
+### Neovim user commands
+
+A convenient way to expose this as editor commands — add to your config after
+`setup()`:
+
+```lua
+-- :RapydLineLength 100
+vim.api.nvim_create_user_command("RapydLineLength", function(args)
+    require("rapydscript").update_settings({ line_length = tonumber(args.args) })
+end, { nargs = 1, desc = "Set RapydScript LSP line length" })
+
+-- :RapydQuote double
+vim.api.nvim_create_user_command("RapydQuote", function(args)
+    require("rapydscript").update_settings({ preferred_quote = args.args })
+end, { nargs = 1, desc = "Set RapydScript LSP preferred quote style" })
+```
+
+### In lazy.nvim config
+
+If you want the commands available automatically, put them in the `config`
+function:
+
+```lua
+{
+    dir = "/path/to/rapydscript/nvim-lsp-plugin",
+    ft = "rapydscript",
+    opts = { line_length = 100 },
+    config = function(_, opts)
+        require("rapydscript").setup(opts)
+
+        vim.api.nvim_create_user_command("RapydLineLength", function(args)
+            require("rapydscript").update_settings({ line_length = tonumber(args.args) })
+        end, { nargs = 1 })
+
+        vim.api.nvim_create_user_command("RapydQuote", function(args)
+            require("rapydscript").update_settings({ preferred_quote = args.args })
+        end, { nargs = 1 })
+    end,
+}
+```
+
+Changes to `line_length` and `preferred_quote` take effect on the next format operation.
+
+## Keymaps
+
+The plugin does not define any keymaps — it relies on the standard Neovim LSP
+keymaps that are set up by `vim.lsp.buf.*`.  A minimal set to add to your LSP
+`on_attach` or a `LspAttach` autocmd:
+
+```lua
+vim.api.nvim_create_autocmd("LspAttach", {
+    callback = function(ev)
+        local buf = ev.buf
+        local map = function(mode, lhs, rhs)
+            vim.keymap.set(mode, lhs, rhs, { buffer = buf })
+        end
+        map("n", "K",           vim.lsp.buf.hover)
+        map("n", "gd",          vim.lsp.buf.definition)
+        map("n", "gr",          vim.lsp.buf.references)
+        map("n", "<leader>rn",  vim.lsp.buf.rename)
+        map("n", "<leader>ca",  vim.lsp.buf.code_action)
+        map("n", "<leader>f",   function() vim.lsp.buf.format({ async = true }) end)
+    end,
+})
+```

package/editor-plugins/nvim/rapydscript/ftdetect/rapydscript.lua

@@ -0,0 +1,10 @@
+#! /usr/bin/env lua
+--
+-- rapydscript.lua
+-- Copyright (C) 2026 Kovid Goyal <kovid at kovidgoyal.net>
+--
+-- Distributed under terms of the MIT license.
+--
+
+
+vim.filetype.add({ extension = { pyj = 'rapydscript' } })

package/editor-plugins/nvim/rapydscript/lua/rapydscript/health.lua

@@ -0,0 +1,88 @@
+local M = {}
+
+local function ver_str(v)
+    return v and table.concat(v, ".") or "unknown"
+end
+
+function M.check()
+    local h = vim.health
+    local rs_ok, rs = pcall(require, "rapydscript")
+    if not rs_ok then
+        h.start("rapydscript")
+        h.error("Failed to load plugin: " .. tostring(rs))
+        return
+    end
+
+    -- Binary -------------------------------------------------------------------
+    h.start("rapydscript binary")
+    local info = rs._binary_info
+    local sys  = info.system
+    local repo = info.repo
+
+    if sys then
+        if sys.rejected then
+            h.error(string.format(
+                "System binary rejected: %s  (v%s < minimum 0.8.0)",
+                sys.path, ver_str(sys.version)
+            ))
+        elseif sys.version then
+            h.ok(string.format("System binary: %s  (v%s)", sys.path, ver_str(sys.version)))
+        else
+            h.warn(string.format("System binary: %s  (version unknown)", sys.path))
+        end
+    else
+        h.warn("rapydscript not found on PATH")
+    end
+
+    if repo then
+        h.ok(string.format("Repo binary:   %s  (v%s)", repo.path, ver_str(repo.version)))
+    else
+        h.info("No repo binary present")
+    end
+
+    if not sys and not repo then
+        h.error("No rapydscript binary found — LSP will not start")
+    end
+
+    -- Tree-sitter --------------------------------------------------------------
+    h.start("rapydscript tree-sitter")
+    if rs._ts_so then
+        h.ok("Parser compiled: " .. rs._ts_so)
+    else
+        h.warn("Tree-sitter parser not compiled (syntax highlighting unavailable)")
+    end
+
+    -- LSP clients --------------------------------------------------------------
+    h.start("rapydscript LSP")
+    local clients = vim.lsp.get_clients({ name = "rapydscript" })
+    if #clients == 0 then
+        h.warn("No active rapydscript LSP client")
+    else
+        for _, client in ipairs(clients) do
+            local cmd_str = type(client.config.cmd) == "table"
+                and table.concat(client.config.cmd, " ")
+                or tostring(client.config.cmd)
+            h.ok(string.format("Client #%d  root: %s", client.id, client.config.root_dir or "?"))
+            h.info("cmd: " .. cmd_str)
+        end
+    end
+
+    -- Settings -----------------------------------------------------------------
+    h.start("rapydscript settings")
+    local opts = rs._active_opts
+    if not opts then
+        h.warn("setup() has not been called — showing defaults")
+        opts = rs.defaults
+    else
+        h.ok("setup() called")
+    end
+
+    h.info("cmd:                    " .. table.concat(opts.cmd, " "))
+    h.info("line_length:            " .. tostring(opts.line_length))
+    h.info("preferred_quote:        " .. tostring(opts.preferred_quote))
+    h.info("filetypes:              " .. table.concat(opts.filetypes, ", "))
+    h.info("root_markers:           " .. table.concat(opts.root_markers, ", "))
+    h.info("import_path_patterns:   " .. table.concat(opts.import_path_patterns, ", "))
+end
+
+return M