execsql2 2.29.0__tar.gz → 2.30.0__tar.gz

execsql2-2.30.0/.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,57 @@
+name: Bug report
+description: Something execsql does wrong, from the command line, the library, or an editor.
+labels: [bug]
+body:
+  - type: textarea
+    id: what
+    attributes:
+      label: What happened
+      description: What you ran, what you expected, and what happened instead.
+      placeholder: |
+        `execsql run load.sql prod.db -t l` stopped at line 12 with ...
+        I expected ...
+    validations:
+      required: true
+  - type: textarea
+    id: script
+    attributes:
+      label: A script that shows it
+      description: The smallest script that reproduces the problem, if you can share one. Leave out data and credentials.
+      render: sql
+  - type: textarea
+    id: output
+    attributes:
+      label: Error output
+      description: The full message or traceback, and the end of `execsql.log` if the run wrote one.
+      render: text
+  - type: input
+    id: version
+    attributes:
+      label: execsql version
+      description: The output of `execsql --version`.
+      placeholder: execsql 2.30.0
+    validations:
+      required: true
+  - type: input
+    id: install
+    attributes:
+      label: How execsql is installed
+      description: The command, with its extras.
+      placeholder: uv tool install "execsql2[postgres,lsp]"
+    validations:
+      required: true
+  - type: input
+    id: environment
+    attributes:
+      label: Operating system, Python and database
+      placeholder: macOS 15, Python 3.12, PostgreSQL 16
+    validations:
+      required: true
+  - type: textarea
+    id: editor
+    attributes:
+      label: Editor and language server (only for a problem in an editor)
+      description: |
+        The editor and the execsql extension's version, the output of `execsql lsp --check`, and the server's log
+        at `info` (VS Code: View → Output → execsql). See https://execsql2.readthedocs.io/en/latest/guides/editors/#troubleshooting
+      render: text

execsql2-2.30.0/.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,28 @@
+name: Feature request
+description: A new feature, metacommand or option, or a change to how something works.
+labels: [enhancement]
+body:
+  - type: textarea
+    id: problem
+    attributes:
+      label: The problem
+      description: What you are trying to do, and what gets in the way today.
+    validations:
+      required: true
+  - type: textarea
+    id: proposal
+    attributes:
+      label: What you would like
+      description: For a new metacommand or CLI option, the syntax you would expect.
+      placeholder: "-- !x! NEW_METACOMMAND ..."
+    validations:
+      required: true
+  - type: textarea
+    id: alternatives
+    attributes:
+      label: Alternatives and workarounds
+      description: What you tried instead, if anything.
+  - type: textarea
+    id: context
+    attributes:
+      label: Anything else

{execsql2-2.29.0 → execsql2-2.30.0}/.github/workflows/ci-cd.yml

@@ -51,25 +51,6 @@ jobs:
             - run: uv sync --locked --all-extras
             - run: uv run --no-sync mypy src/execsql/
 
-    formatter-corpus:
-        name: formatter-corpus
-        runs-on: ubuntu-latest
-        permissions:
-            contents: read
-        steps:
-            - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
-            - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97  # v7.0.0
-              with:
-                  python-version: "3.13"
-                  cache: "pip"
-            - run: python -m pip install --upgrade pip
-            - run: python -m pip install ".[dev]"
-            # Every formatter bug to date came from real SQL containing a
-            # construct the project's own scripts lack.  This gate runs those
-            # checks under their own name so a corpus failure is legible
-            # without reading a 6,000-test log.
-            - run: pytest tests/test_format.py -k corpus --no-cov -q
-
     typer-floor:
         name: typer-floor
         runs-on: ubuntu-latest
@@ -155,7 +136,8 @@ jobs:
             - run: python -m pip install --upgrade pip
             - run: python -m pip install ".[dev]"
             - run: cp CHANGELOG.md docs/about/change_log.md
-            - run: zensical build
+            # --strict: a warning, such as a link to an #anchor that does not exist, fails the job.
+            - run: zensical build --strict
 
     tests:
         strategy:
@@ -369,7 +351,7 @@ jobs:
     build:
         name: Build distribution 📦
         runs-on: ubuntu-latest
-        needs: [ lint, typecheck, pre-commit, formatter-corpus, typer-floor, docs, tests, integration-tests, vscode-extension ]
+        needs: [ lint, typecheck, pre-commit, typer-floor, docs, tests, integration-tests, vscode-extension ]
         # Deliberately NOT gated on tags: packaging breaks must surface on a PR,
         # not at release time. Only publish/generate-release below are tag-gated.
         permissions:

{execsql2-2.29.0 → execsql2-2.30.0}/CHANGELOG.md

@@ -13,6 +13,36 @@ ______________________________________________________________________
 
 ______________________________________________________________________
 
+## [2.30.0] - 2026-10-14
+
+### Added
+
+- **`execsql lsp --log-level` and `--log-file`** (or `EXECSQL_LSP_LOG_LEVEL`): at `info` the server logs its version, the Python running it, whether the formatter is installed, the workspace folders and the config files each folder read; `debug` adds every protocol message. See [Troubleshooting](https://execsql2.readthedocs.io/en/latest/guides/editors/#troubleshooting) for what to include in a bug report.
+- **VS Code extension:** the **execsql** Output channel shows the server's log (setting `execsql.server.logLevel`, default `info`) and why the server did not start; `execsql.trace.server` traces the protocol there.
+
+### Fixed
+
+- **`execsql lint` checks every file** when one has an `INCLUDE` path the operating system rejects (such as a name too long for it): that file gets an I001 finding instead of the whole run stopping with "Uncaught exception", and a script lint cannot check at all is reported as one P001 error.
+- **The language server shows "execsql could not check this script"** when checking a script fails unexpectedly, instead of keeping its old findings; hover, completion and navigation keep working for a script with such an `INCLUDE`.
+- **The language server answers quickly in a script with many or large `INCLUDE`s:** completion, hover, go-to-definition and the other features no longer reread and reparse every included file on each request (50 included 3,000-line files: about 0.55 s per keystroke before, under 20 ms now). A changed included file is read again.
+- **The language server lints less after a save:** saving a script, or a change to a `.sql` file on disk, lints again only the open scripts that `INCLUDE` that file, once, instead of every open script two or three times.
+- **`execsql --help` shows each command's whole summary**, wrapped to the terminal, instead of cutting it off with `...` on a narrow one.
+- **DuckDB temporary tables last for the whole run:** a table created with `CREATE TEMPORARY TABLE` is visible to the statements and metacommands after it (`INSERT`, `EXPORT`, `COPY`, conditions); before, it was gone by the next statement.
+- **A `--dsn` / `dsn=` value without a scheme** (`//user:password@host/db`) is reported with its password as `***` instead of in full.
+- **`--manifest` masks passwords and tokens:** a password, `SUB_DECRYPT` result or token (a URL's password, `password=...`, an API key) that reaches a file path, connection target or error message is written as `***`. `-a` and `--var` values are still written where a path uses them.
+- **A registered secret no longer reaches `execsql.log` through a `HALT MESSAGE`** (the `exit` record of a halted run), an exported file name (`action` records) or a connection name (`connect` records); these now show `***` like every other log record.
+- **`execsql.run()` returns the caller to their working directory** after a script that ran `CD`, including when the script fails; before, the caller was left in the script's last directory.
+- **Language server settings for a workspace folder added after the first folder was deleted** are read; before, every later settings reload failed and the folder linted with stale rules. An unsaved script's relative `INCLUDE` resolves against the workspace root.
+- **VS Code extension:** after `CONFIG`, only options `CONFIG` accepts are highlighted; config-file-only keys such as `DB_FILE`, `CONSOLE_HEIGHT` or `CSS_FILE` (which `CONFIG` rejects) no longer look valid.
+- **VS Code extension:** turning `execsql.server.enabled` on or off, or changing `execsql.server.path`, takes effect without reloading the window; **execsql: Restart Language Server** works (and says so) when the server is off, and two quick restarts no longer leave a second server running.
+
+### Removed
+
+- **Unused helpers:** `execsql.gui.get_backend()` (the GUI picks its backend in `enable_gui()`), `ins_quoted_rx()` (re-exported from `execsql.metacommands`; nothing built a metacommand with it) `execsql.utils.errors.exception_info()` (use `exception_desc()`), and the `execsql.cli.main` re-export (the run command's own module, `execsql.cli.commands.run`, keeps it).
+- **`execsql.utils.fileio`:** `filewriter_closeall()`, `filewriter_shutdown()` and `list_encodings()`, which nothing called; use `filewriter_end()` to stop a writer and `execsql list encodings` for the encodings. `FileWriter.ping()` is gone too.
+
+______________________________________________________________________
+
 ## [2.29.0] - 2026-10-13
 
 ### Changed

{execsql2-2.29.0 → execsql2-2.30.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: execsql2
-Version: 2.29.0
+Version: 2.30.0
 Summary: Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables.
 Project-URL: Homepage, https://execsql2.readthedocs.io
 Project-URL: Repository, https://github.com/geocoug/execsql
@@ -83,7 +83,6 @@ Provides-Extra: auth-plaintext
 Requires-Dist: keyring>=25.0; extra == 'auth-plaintext'
 Requires-Dist: keyrings-alt>=5.0; extra == 'auth-plaintext'
 Provides-Extra: dev
-Requires-Dist: build>=1.2.2.post1; extra == 'dev'
 Requires-Dist: bump-my-version>=1.2.7; extra == 'dev'
 Requires-Dist: defusedxml>=0.7; extra == 'dev'
 Requires-Dist: jinja2>=3.1; extra == 'dev'
@@ -104,7 +103,6 @@ Requires-Dist: ruff>=0.4; extra == 'dev'
 Requires-Dist: sqlglot<30.22,>=30.21.0; extra == 'dev'
 Requires-Dist: tables>=3.10; extra == 'dev'
 Requires-Dist: tox-uv>=1.13.1; extra == 'dev'
-Requires-Dist: twine>=6.1.0; extra == 'dev'
 Requires-Dist: xlrd>=2.0; extra == 'dev'
 Requires-Dist: zensical==0.0.67; extra == 'dev'
 Provides-Extra: duckdb
@@ -466,7 +464,7 @@ execsql format --no-sql --in-place scripts/
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.29.0
+    rev: v2.30.0
     hooks:
       - id: execsql-format
       - id: execsql-lint

{execsql2-2.29.0 → execsql2-2.30.0}/README.md

@@ -321,7 +321,7 @@ execsql format --no-sql --in-place scripts/
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.29.0
+    rev: v2.30.0
     hooks:
       - id: execsql-format
       - id: execsql-lint

{execsql2-2.29.0 → execsql2-2.30.0}/docs/about/divergence.md

@@ -28,7 +28,7 @@ ______________________________________________________________________
 | `--dry-run`                     | Parse the script and print the full command list without connecting to a database or executing anything. Substitution variables already populated at parse time (env vars, `--assign-arg` values, built-in start-time vars) are expanded in the output; execution-time variables (`$DB_NAME`, `$CURRENT_TIME`, etc.) remain unexpanded.            |
 | `--profile`                     | Record wall-clock time for each SQL and metacommand statement. After the script completes, print a summary table sorted by elapsed time (descending), showing time, percentage of total, source location, command type, and a command preview.                                                                                                     |
 | `--profile-limit N`             | Number of top statements to display in the `--profile` summary (default: 20). Remaining statements are counted and noted in the output footer.                                                                                                                                                                                                     |
-| `--manifest FILE`               | Write a JSON record of the run when it ends, even on failure: script, database, statement counts, files read, written and deleted (paths after substitution), connections, errors. Variable values and passwords are never written.                                                                                                                |
+| `--manifest FILE`               | Write a JSON record of the run when it ends, even on failure: script, database, statement counts, files read, written and deleted (paths after substitution), connections, errors. Variable values are never listed; passwords, decrypted values and token shapes are masked as `***` wherever they appear.                                        |
 | `--ping`                        | Test database connectivity and exit. Connects using the supplied connection parameters, queries for the server version, and prints a one-line success message (exit 0) or the error (exit 1). No script file argument is required.                                                                                                                 |
 | `--lint`                        | Removed. Replaced by `execsql lint`, which takes files and directories and adds rule selection and JSON output. `execsql run --lint` now exits 2 with `run --lint was removed; use execsql lint` and never runs the script.                                                                                                                        |
 | `--parse-tree`                  | Parse the script into an Abstract Syntax Tree and print a visual tree showing block nesting (IF/LOOP/BATCH/SCRIPT), source line ranges, compound conditions (ANDIF/ORIF), and all metacommands. Does not connect to a database or execute anything. Useful for understanding script structure and verifying the parser handles a script correctly. |
@@ -205,7 +205,7 @@ result = run(
 result.raise_on_error()
 ```
 
-Accepts `dsn=` or a pre-built `connection=` `Database` object. Each call runs in its own thread-local `RuntimeContext` with its own file-writer thread, so concurrent calls from different threads are safe and a halt in one does not stop another's file output. `halt_on_error=False` collects errors instead of raising.
+Accepts `dsn=` or a pre-built `connection=` `Database` object. Each call runs in its own thread-local `RuntimeContext` with its own file-writer thread, so concurrent calls from different threads are safe and a halt in one does not stop another's file output. `halt_on_error=False` collects errors instead of raising. A script's `CD` lasts until `run()` returns, which then changes back to the caller's directory.
 
 ______________________________________________________________________
 
@@ -267,7 +267,8 @@ All 33 mutable runtime globals in `state.py` have been consolidated into a `Runt
 - **DuckDB temporal types** — `TIMESTAMPTZ`, `TIMESTAMP`, `DATE`, `TIME` now map to native DuckDB types instead of `TEXT`.
 - **SQLite `DT_Long` mapping** — `DT_Long` maps to `"hugeint"` in the SQLite type table. SQLite does not have a native `HUGEINT` type; the value receives `TEXT` affinity. In practice this is harmless because SQLite's type affinity system handles large integers transparently, but the mapping name differs from upstream.
 - **`Database.auto_commits_ddl()` not provided** — 2.18.0 shipped this capability hook on the base `Database` class with `True` overrides on Oracle, MySQL, SQL Server, and MS Access. No call site ever consumed the hook, so 2.19.0 removed it. Its replacement is `Database.implicit_commit_rx`: MySQL and Oracle name the statements their server commits on its own (DDL), and a warning is written when one runs while `AUTOCOMMIT` is `OFF` or a batch is open. SQL Server holds DDL in a transaction like any other statement. (The companion hook `Database.needs_explicit_commit_after_ddl()`, used by the Firebird import path, remains.)
-- **Statements are held until commit on every adapter** — while `AUTOCOMMIT` is `OFF` or a batch is open, `Database.before_statement()` makes sure each statement is inside a transaction. Most drivers already opened one; SQLite's (Python's `sqlite3`) opened one only before `INSERT`/`UPDATE`/`DELETE`/`REPLACE`, and DuckDB's never did, so upstream committed SQLite DDL, and every DuckDB statement, at once. SQLite now begins a transaction before other statements (except `VACUUM`, `ATTACH`, `DETACH`, `PRAGMA`); DuckDB begins one itself, tracks it, and runs statements on the connection rather than on cursors while it is open, since a DuckDB cursor is a separate connection with its own transaction.
+- **Statements are held until commit on every adapter** — while `AUTOCOMMIT` is `OFF` or a batch is open, `Database.before_statement()` makes sure each statement is inside a transaction. Most drivers already opened one; SQLite's (Python's `sqlite3`) opened one only before `INSERT`/`UPDATE`/`DELETE`/`REPLACE`, and DuckDB's never did, so upstream committed SQLite DDL, and every DuckDB statement, at once. SQLite now begins a transaction before other statements (except `VACUUM`, `ATTACH`, `DETACH`, `PRAGMA`); DuckDB begins one itself and tracks it.
+- **DuckDB statements share one connection** — a DuckDB cursor is a separate connection with its own transaction and its own temporary tables, and upstream ran each statement on a new cursor, so a `CREATE TEMPORARY TABLE` was gone by the next statement and a held transaction did not cover them. Every statement now runs on the database's one connection. That connection holds one result at a time, so when a statement runs while another query's rows are still being read (`COPY` scans its source while reading it), the unread rows are fetched into memory first; an `EXPORT` still streams.
 - **DuckDB `$LAST_ROWCOUNT`** — the duckdb driver's `rowcount` is always `-1`; DuckDB answers an `INSERT`, `UPDATE` or `DELETE` with a one-column `Count` result instead. The DuckDB adapter reads that result, so `$LAST_ROWCOUNT` holds the rows changed, as on the other adapters. Upstream set it to `-1`.
 
 ### Execution Engine
@@ -279,7 +280,6 @@ The legacy flat command-list engine (`_parse_script_lines` / `runscripts` / `Com
 - **BREAK outside LOOP is an error** — `BREAK` outside a loop block now raises an error (exit 1) instead of being silently ignored. This catches script bugs that upstream would not report.
 - **File writer is a thread, one per run** — `WRITE`, `TEE` and the other text-file metacommands hand their output to a background thread owned by the run's `RuntimeContext`, which `execsql.run()` stops before returning. Upstream uses one `multiprocessing` subprocess for the whole program; under the `spawn` start method (macOS, Windows) that re-imported the calling program, so a library caller without an `if __name__ == "__main__":` guard ran its script twice.
 - **Instance-scoped script registry** — Named SCRIPT blocks are stored on the `RuntimeContext` instance instead of a module-level dict, preventing cross-execution contamination.
-- **Script source directory resolution** — `ScriptCmd` resolves `source_dir` at construction time (when the command is parsed) rather than per-statement at execution time. This is functionally equivalent for all normal usage since the script file does not move between parse and execution.
 
 ### Error Handling
 
@@ -322,13 +322,13 @@ These are behavioral changes driven by security or correctness issues in the ups
 
 ### Credential and Logging Safety
 
-| Area                          | Fix                                                                                                                                                                                                                                                                                                                                                                               |
-| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| ODBC password redaction       | Connection strings in log output have `Pwd=***` substituted before logging.                                                                                                                                                                                                                                                                                                       |
-| `enc_password` documentation  | Prominent warnings that XOR encryption is obfuscation only — keys are hardcoded in source.                                                                                                                                                                                                                                                                                        |
-| Env-var secret denylist       | Extended `_SENSITIVE_SUBSTRINGS` to cover mainstream cloud / payment / observability / VCS conventions (`*_KEY`, `APIKEY`, `API_KEY`, `DSN`, `WEBHOOK`). `-a` positional assignment log lines now redact unconditionally (`{***}` instead of the raw value).                                                                                                                      |
-| Secrets registered at source  | The `run` record lists a count of `-a` values instead of the values (upstream wrote them in clear). DSN and SMTP passwords, prompted and keyring database passwords, `PROMPT ENTER_SUB ... PASSWORD` and `PROMPT CREDENTIALS` values, and `SUB_DECRYPT` results are registered for redaction, so SQL logged with `LOG_SQL ON`, `SYSTEM_CMD` echoes and error messages show `***`. |
-| SMTP certificate verification | `EMAIL` over `use_ssl` or `use_tls` verifies the server certificate (upstream accepted any certificate, so the SMTP password could go to an impostor). New `[email]` settings: `ca_file` to trust an internal CA, `verify_certificate = No` to turn the check off.                                                                                                                |
+| Area                          | Fix                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
+| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| ODBC password redaction       | Connection strings in log output have `Pwd=***` substituted before logging.                                                                                                                                                                                                                                                                                                                                                                                                   |
+| `enc_password` documentation  | Prominent warnings that XOR encryption is obfuscation only — keys are hardcoded in source.                                                                                                                                                                                                                                                                                                                                                                                    |
+| Env-var secret denylist       | Extended `_SENSITIVE_SUBSTRINGS` to cover mainstream cloud / payment / observability / VCS conventions (`*_KEY`, `APIKEY`, `API_KEY`, `DSN`, `WEBHOOK`). `-a` positional assignment log lines now redact unconditionally (`{***}` instead of the raw value).                                                                                                                                                                                                                  |
+| Secrets registered at source  | The `run` record lists a count of `-a` values instead of the values (upstream wrote them in clear). DSN and SMTP passwords, prompted and keyring database passwords, `PROMPT ENTER_SUB ... PASSWORD` and `PROMPT CREDENTIALS` values, and `SUB_DECRYPT` results are registered for redaction, so SQL logged with `LOG_SQL ON`, `SYSTEM_CMD` echoes, error messages, the `exit` record (including a `HALT MESSAGE`), `action` export records and `connect` records show `***`. |
+| SMTP certificate verification | `EMAIL` over `use_ssl` or `use_tls` verifies the server certificate (upstream accepted any certificate, so the SMTP password could go to an impostor). New `[email]` settings: `ca_file` to trust an internal CA, `verify_certificate = No` to turn the check off.                                                                                                                                                                                                            |
 
 ### Bug Fixes
 

{execsql2-2.29.0 → execsql2-2.30.0}/docs/api/index.md

@@ -80,7 +80,7 @@ Each run has its own [file writer](#file-output), so two runs writing files at t
 
 Three things are still shared by every thread in the process, so keep them apart yourself:
 
-- **The working directory.** `CD` calls `os.chdir()`, which moves every thread. Use absolute paths in scripts that run concurrently, or don't use `CD` in them.
+- **The working directory.** `CD` calls `os.chdir()`, which moves every thread while the script runs. Use absolute paths in scripts that run concurrently, or don't use `CD` in them. When a script that ran `CD` finishes, `run()` changes back to the directory it was in before that `CD`, so the caller ends where it started; a run without `CD` leaves the directory as it finds it.
 - **The same output file.** Two runs writing one file at the same time can interleave their lines. Give each run its own output paths.
 - **Interactive prompts.** `PROMPT` and the other GUI metacommands are not meant to be driven from several threads at once.
 

{execsql2-2.29.0 → execsql2-2.30.0}/docs/dev/architecture.md

@@ -56,6 +56,8 @@ flowchart LR
     UTILS["utils/<br/>auth, crypto,<br/>fileio, mail,<br/>regex, strings"]
     PARSER["parser.py<br/>CondParser,<br/>NumericParser"]
     DEBUG["debug/<br/>REPL debugger"]
+    LSP["lsp/<br/>execsql lsp,<br/>editor features"]
+    REF["metacommands/<br/>reference.py<br/>syntax registry"]
 
     CLI --> CONFIG
     CLI --> STATE
@@ -75,35 +77,66 @@ flowchart LR
     EXPORT --> STATE
     GUI --> STATE
     SCRIPT --> PARSER
+    CLI --> LSP
+    LSP --> SCRIPT
+    LSP --> REF
+    LSP --> CLI
 ```
 
 ### Package summary
 
-| Package          | Purpose                                                                                                                                                         |
-| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `cli/`           | Typer app and help rendering, one module per command under `cli/commands/`, shared options, `_run()` orchestration, DSN URL parsing                             |
-| `api.py`         | Public `execsql.run()` Python entry point for notebooks, pipelines, and library use                                                                             |
-| `config.py`      | `ConfigData` (INI merging), `StatObj` (runtime flags), `WriteHooks` (stdout/stderr redirection)                                                                 |
-| `state.py`       | Thread-local runtime store — all shared mutable state lives here, isolated per-thread                                                                           |
-| `script/`        | AST node types, parser, `MetaCommandList`, `SubVarSet`, `BatchLevels`, `ScriptExecSpec`, `set_system_vars()`                                                    |
-| `metacommands/`  | `build_dispatch_table()`, all `x_*` handlers, `build_conditional_table()`, all `xf_*` predicates                                                                |
-| `db/`            | `Database` ABC, `DatabasePool`, 9 adapter modules (postgres, sqlite, duckdb, mysql, sqlserver, oracle, firebird, access, dsn)                                   |
-| `exporters/`     | `ExportRecord`, `ExportMetadata`, `WriteSpec`, 20+ format writers (CSV, JSON, XML, HTML, etc.)                                                                  |
-| `importers/`     | `CsvFile`, `OdsFile`, `XlsFile`, `FeatherFile` — data import backends                                                                                           |
-| `gui/`           | `GuiBackend` ABC, `TkinterBackend`, `TextualBackend`, `ConsoleBackend`                                                                                          |
-| `utils/`         | Shared utilities: file I/O, encryption, mail, regex helpers, string manipulation, timers                                                                        |
-| `parser.py`      | Recursive-descent parsers for conditional (`IF`) and arithmetic (`SET`) expressions                                                                             |
-| `types.py`       | `DataType` subclasses and `DbType` per-DBMS type dialect mappings                                                                                               |
-| `models.py`      | `Column`, `DataTable`, `JsonDatatype`                                                                                                                           |
-| `format.py`      | `execsql format` formatter — opinionated formatter for execsql scripts                                                                                          |
-| `exceptions.py`  | `ExecSqlError` base, `ErrInfo`, `ConfigError`, `DataTypeError`, `DbTypeError`, etc.                                                                             |
-| `plugins.py`     | Entry-point plugin discovery for metacommands, exporters, and importers                                                                                         |
-| `debug/`         | Interactive REPL debugger for stepping through script execution                                                                                                 |
-| `interactive.py` | Engine behind the debug REPL: input typed at a BREAKPOINT runs as the next lines of the paused script via `executor.execute_input`; errors return to the prompt |
-| `data/`          | Bundled package data (`execsql.conf.template` powers `execsql config --init`)                                                                                   |
+| Package                | Purpose                                                                                                                                                                                        |
+| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `cli/`                 | Typer app and help rendering, one module per command under `cli/commands/`, shared options, `_run()` orchestration, DSN URL parsing                                                            |
+| `api.py`               | Public `execsql.run()` Python entry point for notebooks, pipelines, and library use                                                                                                            |
+| `config.py`            | `ConfigData` (INI merging), `StatObj` (runtime flags), `WriteHooks` (stdout/stderr redirection)                                                                                                |
+| `state.py`             | Thread-local runtime store — all shared mutable state lives here, isolated per-thread                                                                                                          |
+| `script/`              | AST node types, parser, `MetaCommandList`, `SubVarSet`, `BatchLevels`, `ScriptExecSpec`, `set_system_vars()`                                                                                   |
+| `metacommands/`        | `build_dispatch_table()`, all `x_*` handlers, `build_conditional_table()`, all `xf_*` predicates                                                                                               |
+| `db/`                  | `Database` ABC, `DatabasePool`, 9 adapter modules (postgres, sqlite, duckdb, mysql, sqlserver, oracle, firebird, access, dsn)                                                                  |
+| `exporters/`           | `ExportRecord`, `ExportMetadata`, `WriteSpec`, 20+ format writers (CSV, JSON, XML, HTML, etc.)                                                                                                 |
+| `importers/`           | `CsvFile`, `OdsFile`, `XlsFile`, `FeatherFile` — data import backends                                                                                                                          |
+| `gui/`                 | `GuiBackend` ABC, `TkinterBackend`, `TextualBackend`, `ConsoleBackend`                                                                                                                         |
+| `utils/`               | Shared utilities: file I/O, encryption, mail, regex helpers, string manipulation, timers                                                                                                       |
+| `parser.py`            | Recursive-descent parsers for conditional (`IF`) and arithmetic (`SET`) expressions                                                                                                            |
+| `types.py`             | `DataType` subclasses and `DbType` per-DBMS type dialect mappings                                                                                                                              |
+| `models.py`            | `Column`, `DataTable`, `JsonDatatype`                                                                                                                                                          |
+| `format.py`            | `execsql format` formatter — opinionated formatter for execsql scripts                                                                                                                         |
+| `exceptions.py`        | `ExecSqlError` base, `ErrInfo`, `ConfigError`, `DataTypeError`, `DbTypeError`, etc.                                                                                                            |
+| `plugins.py`           | Entry-point plugin discovery for metacommands, exporters, and importers                                                                                                                        |
+| `debug/`               | Interactive REPL debugger for stepping through script execution                                                                                                                                |
+| `interactive.py`       | Engine behind the debug REPL: input typed at a BREAKPOINT runs as the next lines of the paused script via `executor.execute_input`; errors return to the prompt                                |
+| `lsp/`                 | `execsql lsp`, the language server editors start (pygls 2): diagnostics, completion, hover, navigation, rename, folding, quick fixes, Format Document. See [Language Server](#language-server) |
+| `manifest.py`          | `RunManifest`: the JSON record `--manifest FILE` writes when a run ends (script, database, statements, files read/written/deleted, connections, outcome)                                       |
+| `config_validation.py` | `execsql config --validate`: reads the same config files as a run and reports every problem with its file and line                                                                             |
+| `data/`                | Bundled package data (`execsql.conf.template` powers `execsql config --init`)                                                                                                                  |
 
 ______________________________________________________________________
 
+## Language Server { #language-server }
+
+`execsql lsp` (`cli/commands/lsp.py` → `lsp/server.py:create_server()`) is a pygls 2 server on stdin/stdout. It never runs a script or connects to a database: it reuses the static tools.
+
+| Module                                     | Feature                                                 | Reuses                                                             |
+| ------------------------------------------ | ------------------------------------------------------- | ------------------------------------------------------------------ |
+| `server.py`                                | handlers, settings, debounced lint, watched files       | `cli.run._tool_config`                                             |
+| `diagnostics.py`                           | lint findings as LSP diagnostics                        | `cli.lint.lint_source`                                             |
+| `document.py`                              | `ScriptIndex`: definitions, references, `INCLUDE` graph | `script.parser`, `cli.lint.variables_defined_by`                   |
+| `cache.py`                                 | each text's `FileFacts`, included files' text           | `document`                                                         |
+| `_util.py`                                 | `span()`, `markdown()`, the metacommand-line regex      | `lsprotocol`                                                       |
+| `completion.py`, `hover.py`                | completion and hover                                    | `metacommands.reference`                                           |
+| `navigation.py`, `rename.py`, `folding.py` | definition, references, outline, links, rename, folding | `ScriptIndex`, the parser                                          |
+| `code_actions.py`                          | quick fixes for P002, P003, P004, V001                  | the dispatch table (each fix is re-validated before it is offered) |
+| `formatting.py`                            | Format Document                                         | `format.format_file` with the `[format]` settings                  |
+
+**Settings per workspace folder.** `ConfigData(start_dir=...)` (through `_tool_config(None, start_dir=folder)`) reads config as a run started in that directory would: its `execsql.conf`, and relative `config_file` chains resolved against it. `load_settings()` reads each folder's `[lint]` / `[format]` settings that way and keeps them by folder; a document uses its innermost folder's settings, a document in no folder those of the workspace root (`base_dir()`, or the server's working directory when there is no root or it is gone). The server never changes its working directory. An unsaved script's relative `INCLUDE` resolves against `base_dir()` too, for both lint (`lint_source(base_dir=...)`) and the index.
+
+**Index lifetime.** A `ScriptIndex` is assembled on every request that needs one (hover, completion, definition, references, rename, links, code actions) from per-file `FileFacts`: what one file defines, calls, includes and refers to, independent of the files it includes. `lsp/cache.py` keeps each text's facts and each included file's text (by path, size and modification time, dropped when the client reports the file changed), so a request reparses only the edited buffer; a file with a `SUB_INI` is reparsed every time, since its INI file can change on its own. `INCLUDE`d files are read too (the editor's unsaved copy when the file is open), up to 200 files, 10 MiB each and 20 MiB together, regular files only; `SUB_INI` files at most 1 MiB, cached by path, size and modification time.
+
+**Linting.** On open, and 200 ms after the last edit (`LINT_DELAY_SECONDS`), lint runs on the document and the result is published with the document's version. A change to a watched `**/execsql.conf` reloads the settings and relints every open document. Saving a script, or a change to a watched `**/*.sql`, relints only the open documents that are that file or `INCLUDE` it directly: lint reads no included file's text, it only checks that each exists. Both go through the same debounce, so the editor's `didSave` and its watcher's report of the same save lint once.
+
+**Trust.** The server reads the files the workspace names, so the VS Code client (`extras/vscode-execsql/extension.js`) starts it only in a trusted workspace, resolves `execsql` through `PATH` (never from the open folder) and runs it from the home directory. Plugins load from the server's own Python environment, not from the workspace.
+
 ## AST Parser and Executor
 
 execsql2 parses scripts into an AST and walks the tree to execute. There is no separate flat-command-list engine; the AST executor is the only path.
@@ -157,6 +190,10 @@ ______________________________________________________________________
 
 Metacommands are lines in SQL scripts prefixed with `-- !x!`. At import time, `metacommands/__init__.py` calls `build_dispatch_table()`, which populates a `MetaCommandList` with all `mcl.add()` registrations (~225 regex patterns). This singleton is stored as `_state.metacommandlist`.
 
+### The syntax registry
+
+`metacommands/reference.py` describes every metacommand a second way: the syntax forms as the reference docs write them, a one-line summary, a category and the docs anchor. The regex dispatch table decides what *runs*; the registry is what people and editors *read*. `execsql list metacommands`, the language server's completion snippets and hover text come from it. `tests/test_reference.py` checks that every registry form is accepted by the dispatch table, that every dispatch keyword has an entry, and that each anchor exists in `docs/reference/metacommands.md`. A new metacommand therefore needs both: an `mcl.add()` registration and a registry entry.
+
 ### How dispatch works
 
 For each `MetaCommandStatement` AST node, the executor calls `_exec_metacommand()` in `script/executor.py`, which delegates to `_state.metacommandlist.eval(cmd_str)`. The dispatcher extracts the leading keyword, narrows ~225 entries to a small candidate set via a keyword index, then tests each candidate's compiled regex against the full command. The matched handler is called with the regex's named groups as keyword arguments plus `metacommandline` (the original unmodified line), and its hit counter is incremented.
@@ -303,19 +340,19 @@ This import pattern (always as `_state`, always accessed inside function/method
 
 ### Key state variables
 
-| Variable          | Type                     | Purpose                                                          |
-| ----------------- | ------------------------ | ---------------------------------------------------------------- |
-| `conf`            | `ConfigData`             | Merged configuration                                             |
-| `subvars`         | `SubVarSet`              | Global substitution variables                                    |
-| `ast_exec_stack`  | `list[ExecFrame]`        | Unified execution stack (scopes + IF/LOOP/BATCH/INCLUDE)         |
-| `ast_scripts`     | `dict[str, ScriptBlock]` | Named scripts from `BEGIN/END SCRIPT` (AST registry)             |
-| `last_command`    | `ScriptCmd \| None`      | Most-recently-executed statement; powers `current_script_line()` |
-| `dbs`             | `DatabasePool`           | Open database connections                                        |
-| `metacommandlist` | `MetaCommandList`        | Metacommand dispatch table                                       |
-| `conditionallist` | `MetaCommandList`        | Conditional predicate dispatch table                             |
-| `counters`        | `CounterVars`            | Auto-incrementing counters                                       |
-| `filewriter`      | `FileWriter`             | This run's background file-writing thread                        |
-| `exec_log`        | `Logger`                 | Execution log                                                    |
+| Variable          | Type                         | Purpose                                                          |
+| ----------------- | ---------------------------- | ---------------------------------------------------------------- |
+| `conf`            | `ConfigData`                 | Merged configuration                                             |
+| `subvars`         | `SubVarSet`                  | Global substitution variables                                    |
+| `ast_exec_stack`  | `list[ExecFrame]`            | Unified execution stack (scopes + IF/LOOP/BATCH/INCLUDE)         |
+| `ast_scripts`     | `dict[str, ScriptBlock]`     | Named scripts from `BEGIN/END SCRIPT` (AST registry)             |
+| `last_command`    | `ExecutingStatement \| None` | Most-recently-executed statement; powers `current_script_line()` |
+| `dbs`             | `DatabasePool`               | Open database connections                                        |
+| `metacommandlist` | `MetaCommandList`            | Metacommand dispatch table                                       |
+| `conditionallist` | `MetaCommandList`            | Conditional predicate dispatch table                             |
+| `counters`        | `CounterVars`                | Auto-incrementing counters                                       |
+| `filewriter`      | `FileWriter`                 | This run's background file-writing thread                        |
+| `exec_log`        | `Logger`                     | Execution log                                                    |
 
 ______________________________________________________________________
 
@@ -328,7 +365,7 @@ ______________________________________________________________________
 1. System-wide (platform-dependent)
 1. User home directory
 1. Script file directory
-1. Current working directory
+1. Current working directory (or `start_dir`, when the caller passes one)
 
 It exposes all recognized options as attributes (`db_type`, `server`, `db`, `db_file`, `username`, `script_encoding`, `output_encoding`, etc.).
 

{execsql2-2.29.0 → execsql2-2.30.0}/docs/getting-started/syntax.md

@@ -300,8 +300,8 @@ The options below belong to `run`: `execsql run -tl script.sql mydb.sqlite`.
 `--version` and `-o`/`--online-help` work on their own (`execsql --version`)
 and anywhere on a `run` command line.
 `format` has its own options, described in the [formatter guide](../guides/formatter.md);
-`lint`'s are in the [lint rules reference](../reference/lint.md); `lsp`'s (`--check`) in
-[Editor Support](../guides/editors.md#install); `config`, `list` and `init` are described
+`lint`'s are in the [lint rules reference](../reference/lint.md); `lsp`'s (`--check`,
+`--log-level`, `--log-file`) in [Editor Support](../guides/editors.md#troubleshooting); `config`, `list` and `init` are described
 [above](#config_command).
 
 ### Connection options
@@ -479,7 +479,7 @@ Valid encoding names can be displayed with `execsql list encodings`. See also [C
     }
     ```
 
-    Files and connections are recorded as each metacommand succeeds, after variable substitution, so paths are the ones used (`out/west.csv`, not `out/!!region!!.csv`); `INCLUDE`d scripts count as files read. When a run fails, `exit_status` is its exit code and `errors` holds the message, its `type`, `source`, `line` and the failing `command`. `variables` lists the names set with `-a` and `--var`; their values are never written, for the same reason the log hides them, and neither are passwords.
+    Files and connections are recorded as each metacommand succeeds, after variable substitution, so paths are the ones used (`out/west.csv`, not `out/!!region!!.csv`); `INCLUDE`d scripts count as files read. When a run fails, `exit_status` is its exit code and `errors` holds the message, its `type`, `source`, `line` and the failing `command`. `variables` lists the names set with `-a` and `--var`, not their values. A password (from `--dsn`, a prompt or the keyring), a `SUB_DECRYPT` result, or a common secret shape (a URL's password, `password=...`, an API token) that ends up in a path, a connection target or an error message is written as `***`. Values given with `-a` and `--var` are kept where a path uses them, as in `out/west.csv`, although the log masks them: keep a secret out of `-a` and `--var` if the manifest is collected.
 
     The path is checked before the run starts: if *FILE* cannot be written (its folder is a file, or not writable), `execsql run` exits 2 before any SQL runs.
 

{execsql2-2.29.0 → execsql2-2.30.0}/docs/guides/editors.md

@@ -101,7 +101,7 @@ underlined line). A fix is offered only when applying it clears the finding.
 Format Document (VS Code: Shift-Alt-F, or `editor.formatOnSave`) runs
 [`execsql format`](formatter.md) on the editor's text, with the
 `[format]` settings of the workspace's configuration (`indent`, `sql`,
-`leading_comma`), so the editor, the command line and the `execsql-format`
+`leading_comma`, and [`rewrite_sql`](formatter.md#layout-only)), so the editor, the command line and the `execsql-format`
 pre-commit hook agree. The editor's own tab-size setting is not used. Only
 whole documents are formatted, not selections: a statement's indentation
 depends on the IF, LOOP and SCRIPT blocks around it.
@@ -148,13 +148,18 @@ loads/                    (no execsql.conf)
 an unknown metacommand is reported in `loads/*.sql` but not in
 `reports/*.sql`. Adding or removing a folder takes effect at once.
 
-The server reads each folder's `execsql.conf` again when it changes, and
-lints every open script again when a `.sql` file in the workspace changes or
-is saved, so a script sees the variables an edited `INCLUDE` file now defines.
-An included file that is open in the editor is read as you have it, saved or
-not. This needs an editor that reports file changes to the server (VS Code and
-Neovim do); otherwise, and after changing a configuration file outside the
-workspace, restart the server.
+The server reads each folder's `execsql.conf` again when it changes (and lints
+every open script with the new settings), and lints an open script again when a
+file it `INCLUDE`s is created, deleted or saved, so a missing-file finding
+clears once the file exists. Completion, hover and go-to-definition always use
+the included files' current text, so a script sees the variables an edited
+`INCLUDE` file now defines. An included file that is open in the editor is read
+as you have it, saved or not. This needs an editor that reports file changes to the server: VS Code
+does, and Neovim does on macOS and Windows. Elsewhere (Neovim on Linux, most
+other clients) saving a script in the editor still relints the scripts that
+include it, but an edit to `execsql.conf`, or a change made outside the editor
+(a `git checkout`, another tool), is picked up only after restarting the server
+(Neovim: `:LspRestart`).
 
 The server only reads: it never connects to a database or runs a script, and
 nothing in the workspace can make it run code. The only code it loads besides
@@ -182,7 +187,8 @@ The setting is a command name looked up in PATH or an absolute path; a relative
 path is refused. The extension never looks for `execsql` in the open folder,
 and starts the server from your home directory.
 After upgrading execsql, run **execsql: Restart Language Server** from the
-Command Palette. The extension also highlights execsql syntax; that works
+Command Palette. The server's log is in **View → Output → execsql**
+([Troubleshooting](#troubleshooting)). The extension also highlights execsql syntax; that works
 without the server.
 
 In a folder you have not marked as trusted
@@ -227,3 +233,45 @@ Any editor with a language server client can use it: configure a server for
 the `sql` language whose command is `execsql lsp`, run over stdio. Sublime Text
 (the LSP package), Emacs (eglot or lsp-mode), Zed and JetBrains IDEs (the LSP4IJ
 plugin) all work this way.
+
+## Troubleshooting { #troubleshooting }
+
+**Where the server's log is.** The server writes its log to stderr, which the
+editor keeps:
+
+| Editor  | Where                                                                                 |
+| ------- | ------------------------------------------------------------------------------------- |
+| VS Code | **View → Output**, then choose **execsql** in the list. Start failures are there too. |
+| Neovim  | `:LspLog` (`~/.local/state/nvim/lsp.log`)                                             |
+| Helix   | `:log-open`                                                                           |
+
+**How much it logs.** `execsql lsp --log-level` (or the `EXECSQL_LSP_LOG_LEVEL`
+environment variable) takes `error`, `warning` (the default), `info` or `debug`:
+
+- `info` adds a startup record and each workspace folder's settings, which is
+    what a bug report needs:
+
+    ```text
+    INFO execsql.lsp.server: execsql lsp 2.x.y; Python 3.12.7 at /home/me/.local/share/uv/tools/execsql2/bin/python; pygls 2.x.y; formatter available
+    INFO execsql.lsp.server: workspace folders: /home/me/etl
+    INFO execsql.lsp.server: settings for /home/me/etl: config files /home/me/etl/execsql.conf; lint select all, ignore V; format {...}
+    ```
+
+- `debug` adds every message between the editor and the server, including the
+    scripts' text.
+
+The VS Code extension logs at `info` (setting **execsql.server.logLevel**). In
+another editor, add the option to the server command, e.g. in Neovim
+`cmd = { "execsql", "lsp", "--log-level", "info" }`. An editor that does not
+keep the server's stderr can use `--log-file PATH` to append the log to a file.
+
+**Tracing the protocol in VS Code.** Set **execsql.trace.server** to `messages`
+(or `verbose`, which adds each message's content); the trace appears in the same
+Output channel.
+
+**Reporting a problem.** [Open an issue](https://github.com/geocoug/execsql/issues)
+with:
+
+1. The output of `execsql --version` and `execsql lsp --check`.
+1. The server log at `info`, from start-up to the problem.
+1. A small script that shows it, if you can share one.