execsql2 2.28.0__tar.gz → 2.30.0__tar.gz

execsql2-2.30.0/.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,57 @@
+name: Bug report
+description: Something execsql does wrong, from the command line, the library, or an editor.
+labels: [bug]
+body:
+  - type: textarea
+    id: what
+    attributes:
+      label: What happened
+      description: What you ran, what you expected, and what happened instead.
+      placeholder: |
+        `execsql run load.sql prod.db -t l` stopped at line 12 with ...
+        I expected ...
+    validations:
+      required: true
+  - type: textarea
+    id: script
+    attributes:
+      label: A script that shows it
+      description: The smallest script that reproduces the problem, if you can share one. Leave out data and credentials.
+      render: sql
+  - type: textarea
+    id: output
+    attributes:
+      label: Error output
+      description: The full message or traceback, and the end of `execsql.log` if the run wrote one.
+      render: text
+  - type: input
+    id: version
+    attributes:
+      label: execsql version
+      description: The output of `execsql --version`.
+      placeholder: execsql 2.30.0
+    validations:
+      required: true
+  - type: input
+    id: install
+    attributes:
+      label: How execsql is installed
+      description: The command, with its extras.
+      placeholder: uv tool install "execsql2[postgres,lsp]"
+    validations:
+      required: true
+  - type: input
+    id: environment
+    attributes:
+      label: Operating system, Python and database
+      placeholder: macOS 15, Python 3.12, PostgreSQL 16
+    validations:
+      required: true
+  - type: textarea
+    id: editor
+    attributes:
+      label: Editor and language server (only for a problem in an editor)
+      description: |
+        The editor and the execsql extension's version, the output of `execsql lsp --check`, and the server's log
+        at `info` (VS Code: View → Output → execsql). See https://execsql2.readthedocs.io/en/latest/guides/editors/#troubleshooting
+      render: text

execsql2-2.30.0/.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,28 @@
+name: Feature request
+description: A new feature, metacommand or option, or a change to how something works.
+labels: [enhancement]
+body:
+  - type: textarea
+    id: problem
+    attributes:
+      label: The problem
+      description: What you are trying to do, and what gets in the way today.
+    validations:
+      required: true
+  - type: textarea
+    id: proposal
+    attributes:
+      label: What you would like
+      description: For a new metacommand or CLI option, the syntax you would expect.
+      placeholder: "-- !x! NEW_METACOMMAND ..."
+    validations:
+      required: true
+  - type: textarea
+    id: alternatives
+    attributes:
+      label: Alternatives and workarounds
+      description: What you tried instead, if anything.
+  - type: textarea
+    id: context
+    attributes:
+      label: Anything else

{execsql2-2.28.0 → execsql2-2.30.0}/.github/workflows/ci-cd.yml

@@ -51,25 +51,6 @@ jobs:
             - run: uv sync --locked --all-extras
             - run: uv run --no-sync mypy src/execsql/
 
-    formatter-corpus:
-        name: formatter-corpus
-        runs-on: ubuntu-latest
-        permissions:
-            contents: read
-        steps:
-            - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
-            - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97  # v7.0.0
-              with:
-                  python-version: "3.13"
-                  cache: "pip"
-            - run: python -m pip install --upgrade pip
-            - run: python -m pip install ".[dev]"
-            # Every formatter bug to date came from real SQL containing a
-            # construct the project's own scripts lack.  This gate runs those
-            # checks under their own name so a corpus failure is legible
-            # without reading a 6,000-test log.
-            - run: pytest tests/test_format.py -k corpus --no-cov -q
-
     typer-floor:
         name: typer-floor
         runs-on: ubuntu-latest
@@ -155,7 +136,8 @@ jobs:
             - run: python -m pip install --upgrade pip
             - run: python -m pip install ".[dev]"
             - run: cp CHANGELOG.md docs/about/change_log.md
-            - run: zensical build
+            # --strict: a warning, such as a link to an #anchor that does not exist, fails the job.
+            - run: zensical build --strict
 
     tests:
         strategy:
@@ -369,7 +351,7 @@ jobs:
     build:
         name: Build distribution 📦
         runs-on: ubuntu-latest
-        needs: [ lint, typecheck, pre-commit, formatter-corpus, typer-floor, docs, tests, integration-tests, vscode-extension ]
+        needs: [ lint, typecheck, pre-commit, typer-floor, docs, tests, integration-tests, vscode-extension ]
         # Deliberately NOT gated on tags: packaging breaks must surface on a PR,
         # not at release time. Only publish/generate-release below are tag-gated.
         permissions:

{execsql2-2.28.0 → execsql2-2.30.0}/CHANGELOG.md

@@ -13,6 +13,51 @@ ______________________________________________________________________
 
 ______________________________________________________________________
 
+## [2.30.0] - 2026-10-14
+
+### Added
+
+- **`execsql lsp --log-level` and `--log-file`** (or `EXECSQL_LSP_LOG_LEVEL`): at `info` the server logs its version, the Python running it, whether the formatter is installed, the workspace folders and the config files each folder read; `debug` adds every protocol message. See [Troubleshooting](https://execsql2.readthedocs.io/en/latest/guides/editors/#troubleshooting) for what to include in a bug report.
+- **VS Code extension:** the **execsql** Output channel shows the server's log (setting `execsql.server.logLevel`, default `info`) and why the server did not start; `execsql.trace.server` traces the protocol there.
+
+### Fixed
+
+- **`execsql lint` checks every file** when one has an `INCLUDE` path the operating system rejects (such as a name too long for it): that file gets an I001 finding instead of the whole run stopping with "Uncaught exception", and a script lint cannot check at all is reported as one P001 error.
+- **The language server shows "execsql could not check this script"** when checking a script fails unexpectedly, instead of keeping its old findings; hover, completion and navigation keep working for a script with such an `INCLUDE`.
+- **The language server answers quickly in a script with many or large `INCLUDE`s:** completion, hover, go-to-definition and the other features no longer reread and reparse every included file on each request (50 included 3,000-line files: about 0.55 s per keystroke before, under 20 ms now). A changed included file is read again.
+- **The language server lints less after a save:** saving a script, or a change to a `.sql` file on disk, lints again only the open scripts that `INCLUDE` that file, once, instead of every open script two or three times.
+- **`execsql --help` shows each command's whole summary**, wrapped to the terminal, instead of cutting it off with `...` on a narrow one.
+- **DuckDB temporary tables last for the whole run:** a table created with `CREATE TEMPORARY TABLE` is visible to the statements and metacommands after it (`INSERT`, `EXPORT`, `COPY`, conditions); before, it was gone by the next statement.
+- **A `--dsn` / `dsn=` value without a scheme** (`//user:password@host/db`) is reported with its password as `***` instead of in full.
+- **`--manifest` masks passwords and tokens:** a password, `SUB_DECRYPT` result or token (a URL's password, `password=...`, an API key) that reaches a file path, connection target or error message is written as `***`. `-a` and `--var` values are still written where a path uses them.
+- **A registered secret no longer reaches `execsql.log` through a `HALT MESSAGE`** (the `exit` record of a halted run), an exported file name (`action` records) or a connection name (`connect` records); these now show `***` like every other log record.
+- **`execsql.run()` returns the caller to their working directory** after a script that ran `CD`, including when the script fails; before, the caller was left in the script's last directory.
+- **Language server settings for a workspace folder added after the first folder was deleted** are read; before, every later settings reload failed and the folder linted with stale rules. An unsaved script's relative `INCLUDE` resolves against the workspace root.
+- **VS Code extension:** after `CONFIG`, only options `CONFIG` accepts are highlighted; config-file-only keys such as `DB_FILE`, `CONSOLE_HEIGHT` or `CSS_FILE` (which `CONFIG` rejects) no longer look valid.
+- **VS Code extension:** turning `execsql.server.enabled` on or off, or changing `execsql.server.path`, takes effect without reloading the window; **execsql: Restart Language Server** works (and says so) when the server is off, and two quick restarts no longer leave a second server running.
+
+### Removed
+
+- **Unused helpers:** `execsql.gui.get_backend()` (the GUI picks its backend in `enable_gui()`), `ins_quoted_rx()` (re-exported from `execsql.metacommands`; nothing built a metacommand with it) `execsql.utils.errors.exception_info()` (use `exception_desc()`), and the `execsql.cli.main` re-export (the run command's own module, `execsql.cli.commands.run`, keeps it).
+- **`execsql.utils.fileio`:** `filewriter_closeall()`, `filewriter_shutdown()` and `list_encodings()`, which nothing called; use `filewriter_end()` to stop a writer and `execsql list encodings` for the encodings. `FileWriter.ping()` is gone too.
+
+______________________________________________________________________
+
+## [2.29.0] - 2026-10-13
+
+### Changed
+
+- **`!'!var!'!` stores backslashes as written** on every DBMS: a value such as `C:\data\new` was stored as `C:\\data\\new` on SQLite, PostgreSQL, DuckDB, SQL Server, Oracle, Firebird and Access. MySQL/MariaDB (which read `\` as an escape) and PostgreSQL escape strings written as `E!'!var!'!` still get backslashes doubled, so both forms store the same value and the `E` workaround is no longer needed. Scripts that removed the doubled backslashes some other way should stop.
+
+### Fixed
+
+- **Oracle IMPORT and COPY with a time column** create it as text (`VARCHAR2(15)`, holding `HH:MM:SS`); `TO NEW` stopped with "Oracle DBMS type has no specification for data type time".
+- **MySQL IMPORT accepts a header with capital letters** (`Id,Name`): it stopped with "columns that are not in table" (leaving the new table behind), and with `IMPORT_COMMON_COLUMNS_ONLY` it loaded those columns as NULL.
+- **PostgreSQL IMPORT honors `SKIP n`** for quoted CSV files, which load through PostgreSQL's COPY: it skipped only the header line, loading the lines before it as data or failing on them.
+- **Spreadsheet IMPORT reads every data row:** an XLSX row whose values are all `0` or FALSE no longer ends the data (it and every row after it were dropped), and an XLS sheet imported with `SKIP n` no longer loses its last *n* rows.
+
+______________________________________________________________________
+
 ## [2.28.0] - 2026-10-13
 
 ### Added

{execsql2-2.28.0 → execsql2-2.30.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: execsql2
-Version: 2.28.0
+Version: 2.30.0
 Summary: Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables.
 Project-URL: Homepage, https://execsql2.readthedocs.io
 Project-URL: Repository, https://github.com/geocoug/execsql
@@ -83,7 +83,6 @@ Provides-Extra: auth-plaintext
 Requires-Dist: keyring>=25.0; extra == 'auth-plaintext'
 Requires-Dist: keyrings-alt>=5.0; extra == 'auth-plaintext'
 Provides-Extra: dev
-Requires-Dist: build>=1.2.2.post1; extra == 'dev'
 Requires-Dist: bump-my-version>=1.2.7; extra == 'dev'
 Requires-Dist: defusedxml>=0.7; extra == 'dev'
 Requires-Dist: jinja2>=3.1; extra == 'dev'
@@ -104,7 +103,6 @@ Requires-Dist: ruff>=0.4; extra == 'dev'
 Requires-Dist: sqlglot<30.22,>=30.21.0; extra == 'dev'
 Requires-Dist: tables>=3.10; extra == 'dev'
 Requires-Dist: tox-uv>=1.13.1; extra == 'dev'
-Requires-Dist: twine>=6.1.0; extra == 'dev'
 Requires-Dist: xlrd>=2.0; extra == 'dev'
 Requires-Dist: zensical==0.0.67; extra == 'dev'
 Provides-Extra: duckdb
@@ -466,7 +464,7 @@ execsql format --no-sql --in-place scripts/
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.28.0
+    rev: v2.30.0
     hooks:
       - id: execsql-format
       - id: execsql-lint

{execsql2-2.28.0 → execsql2-2.30.0}/README.md

@@ -321,7 +321,7 @@ execsql format --no-sql --in-place scripts/
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.28.0
+    rev: v2.30.0
     hooks:
       - id: execsql-format
       - id: execsql-lint

{execsql2-2.28.0 → execsql2-2.30.0}/docs/about/divergence.md

@@ -28,7 +28,7 @@ ______________________________________________________________________
 | `--dry-run`                     | Parse the script and print the full command list without connecting to a database or executing anything. Substitution variables already populated at parse time (env vars, `--assign-arg` values, built-in start-time vars) are expanded in the output; execution-time variables (`$DB_NAME`, `$CURRENT_TIME`, etc.) remain unexpanded.            |
 | `--profile`                     | Record wall-clock time for each SQL and metacommand statement. After the script completes, print a summary table sorted by elapsed time (descending), showing time, percentage of total, source location, command type, and a command preview.                                                                                                     |
 | `--profile-limit N`             | Number of top statements to display in the `--profile` summary (default: 20). Remaining statements are counted and noted in the output footer.                                                                                                                                                                                                     |
-| `--manifest FILE`               | Write a JSON record of the run when it ends, even on failure: script, database, statement counts, files read, written and deleted (paths after substitution), connections, errors. Variable values and passwords are never written.                                                                                                                |
+| `--manifest FILE`               | Write a JSON record of the run when it ends, even on failure: script, database, statement counts, files read, written and deleted (paths after substitution), connections, errors. Variable values are never listed; passwords, decrypted values and token shapes are masked as `***` wherever they appear.                                        |
 | `--ping`                        | Test database connectivity and exit. Connects using the supplied connection parameters, queries for the server version, and prints a one-line success message (exit 0) or the error (exit 1). No script file argument is required.                                                                                                                 |
 | `--lint`                        | Removed. Replaced by `execsql lint`, which takes files and directories and adds rule selection and JSON output. `execsql run --lint` now exits 2 with `run --lint was removed; use execsql lint` and never runs the script.                                                                                                                        |
 | `--parse-tree`                  | Parse the script into an Abstract Syntax Tree and print a visual tree showing block nesting (IF/LOOP/BATCH/SCRIPT), source line ranges, compound conditions (ANDIF/ORIF), and all metacommands. Does not connect to a database or execute anything. Useful for understanding script structure and verifying the parser handles a script correctly. |
@@ -205,7 +205,7 @@ result = run(
 result.raise_on_error()
 ```
 
-Accepts `dsn=` or a pre-built `connection=` `Database` object. Each call runs in its own thread-local `RuntimeContext` with its own file-writer thread, so concurrent calls from different threads are safe and a halt in one does not stop another's file output. `halt_on_error=False` collects errors instead of raising.
+Accepts `dsn=` or a pre-built `connection=` `Database` object. Each call runs in its own thread-local `RuntimeContext` with its own file-writer thread, so concurrent calls from different threads are safe and a halt in one does not stop another's file output. `halt_on_error=False` collects errors instead of raising. A script's `CD` lasts until `run()` returns, which then changes back to the caller's directory.
 
 ______________________________________________________________________
 
@@ -267,7 +267,8 @@ All 33 mutable runtime globals in `state.py` have been consolidated into a `Runt
 - **DuckDB temporal types** — `TIMESTAMPTZ`, `TIMESTAMP`, `DATE`, `TIME` now map to native DuckDB types instead of `TEXT`.
 - **SQLite `DT_Long` mapping** — `DT_Long` maps to `"hugeint"` in the SQLite type table. SQLite does not have a native `HUGEINT` type; the value receives `TEXT` affinity. In practice this is harmless because SQLite's type affinity system handles large integers transparently, but the mapping name differs from upstream.
 - **`Database.auto_commits_ddl()` not provided** — 2.18.0 shipped this capability hook on the base `Database` class with `True` overrides on Oracle, MySQL, SQL Server, and MS Access. No call site ever consumed the hook, so 2.19.0 removed it. Its replacement is `Database.implicit_commit_rx`: MySQL and Oracle name the statements their server commits on its own (DDL), and a warning is written when one runs while `AUTOCOMMIT` is `OFF` or a batch is open. SQL Server holds DDL in a transaction like any other statement. (The companion hook `Database.needs_explicit_commit_after_ddl()`, used by the Firebird import path, remains.)
-- **Statements are held until commit on every adapter** — while `AUTOCOMMIT` is `OFF` or a batch is open, `Database.before_statement()` makes sure each statement is inside a transaction. Most drivers already opened one; SQLite's (Python's `sqlite3`) opened one only before `INSERT`/`UPDATE`/`DELETE`/`REPLACE`, and DuckDB's never did, so upstream committed SQLite DDL, and every DuckDB statement, at once. SQLite now begins a transaction before other statements (except `VACUUM`, `ATTACH`, `DETACH`, `PRAGMA`); DuckDB begins one itself, tracks it, and runs statements on the connection rather than on cursors while it is open, since a DuckDB cursor is a separate connection with its own transaction.
+- **Statements are held until commit on every adapter** — while `AUTOCOMMIT` is `OFF` or a batch is open, `Database.before_statement()` makes sure each statement is inside a transaction. Most drivers already opened one; SQLite's (Python's `sqlite3`) opened one only before `INSERT`/`UPDATE`/`DELETE`/`REPLACE`, and DuckDB's never did, so upstream committed SQLite DDL, and every DuckDB statement, at once. SQLite now begins a transaction before other statements (except `VACUUM`, `ATTACH`, `DETACH`, `PRAGMA`); DuckDB begins one itself and tracks it.
+- **DuckDB statements share one connection** — a DuckDB cursor is a separate connection with its own transaction and its own temporary tables, and upstream ran each statement on a new cursor, so a `CREATE TEMPORARY TABLE` was gone by the next statement and a held transaction did not cover them. Every statement now runs on the database's one connection. That connection holds one result at a time, so when a statement runs while another query's rows are still being read (`COPY` scans its source while reading it), the unread rows are fetched into memory first; an `EXPORT` still streams.
 - **DuckDB `$LAST_ROWCOUNT`** — the duckdb driver's `rowcount` is always `-1`; DuckDB answers an `INSERT`, `UPDATE` or `DELETE` with a one-column `Count` result instead. The DuckDB adapter reads that result, so `$LAST_ROWCOUNT` holds the rows changed, as on the other adapters. Upstream set it to `-1`.
 
 ### Execution Engine
@@ -279,7 +280,6 @@ The legacy flat command-list engine (`_parse_script_lines` / `runscripts` / `Com
 - **BREAK outside LOOP is an error** — `BREAK` outside a loop block now raises an error (exit 1) instead of being silently ignored. This catches script bugs that upstream would not report.
 - **File writer is a thread, one per run** — `WRITE`, `TEE` and the other text-file metacommands hand their output to a background thread owned by the run's `RuntimeContext`, which `execsql.run()` stops before returning. Upstream uses one `multiprocessing` subprocess for the whole program; under the `spawn` start method (macOS, Windows) that re-imported the calling program, so a library caller without an `if __name__ == "__main__":` guard ran its script twice.
 - **Instance-scoped script registry** — Named SCRIPT blocks are stored on the `RuntimeContext` instance instead of a module-level dict, preventing cross-execution contamination.
-- **Script source directory resolution** — `ScriptCmd` resolves `source_dir` at construction time (when the command is parsed) rather than per-statement at execution time. This is functionally equivalent for all normal usage since the script file does not move between parse and execution.
 
 ### Error Handling
 
@@ -322,13 +322,13 @@ These are behavioral changes driven by security or correctness issues in the ups
 
 ### Credential and Logging Safety
 
-| Area                          | Fix                                                                                                                                                                                                                                                                                                                                                                               |
-| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| ODBC password redaction       | Connection strings in log output have `Pwd=***` substituted before logging.                                                                                                                                                                                                                                                                                                       |
-| `enc_password` documentation  | Prominent warnings that XOR encryption is obfuscation only — keys are hardcoded in source.                                                                                                                                                                                                                                                                                        |
-| Env-var secret denylist       | Extended `_SENSITIVE_SUBSTRINGS` to cover mainstream cloud / payment / observability / VCS conventions (`*_KEY`, `APIKEY`, `API_KEY`, `DSN`, `WEBHOOK`). `-a` positional assignment log lines now redact unconditionally (`{***}` instead of the raw value).                                                                                                                      |
-| Secrets registered at source  | The `run` record lists a count of `-a` values instead of the values (upstream wrote them in clear). DSN and SMTP passwords, prompted and keyring database passwords, `PROMPT ENTER_SUB ... PASSWORD` and `PROMPT CREDENTIALS` values, and `SUB_DECRYPT` results are registered for redaction, so SQL logged with `LOG_SQL ON`, `SYSTEM_CMD` echoes and error messages show `***`. |
-| SMTP certificate verification | `EMAIL` over `use_ssl` or `use_tls` verifies the server certificate (upstream accepted any certificate, so the SMTP password could go to an impostor). New `[email]` settings: `ca_file` to trust an internal CA, `verify_certificate = No` to turn the check off.                                                                                                                |
+| Area                          | Fix                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
+| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| ODBC password redaction       | Connection strings in log output have `Pwd=***` substituted before logging.                                                                                                                                                                                                                                                                                                                                                                                                   |
+| `enc_password` documentation  | Prominent warnings that XOR encryption is obfuscation only — keys are hardcoded in source.                                                                                                                                                                                                                                                                                                                                                                                    |
+| Env-var secret denylist       | Extended `_SENSITIVE_SUBSTRINGS` to cover mainstream cloud / payment / observability / VCS conventions (`*_KEY`, `APIKEY`, `API_KEY`, `DSN`, `WEBHOOK`). `-a` positional assignment log lines now redact unconditionally (`{***}` instead of the raw value).                                                                                                                                                                                                                  |
+| Secrets registered at source  | The `run` record lists a count of `-a` values instead of the values (upstream wrote them in clear). DSN and SMTP passwords, prompted and keyring database passwords, `PROMPT ENTER_SUB ... PASSWORD` and `PROMPT CREDENTIALS` values, and `SUB_DECRYPT` results are registered for redaction, so SQL logged with `LOG_SQL ON`, `SYSTEM_CMD` echoes, error messages, the `exit` record (including a `HALT MESSAGE`), `action` export records and `connect` records show `***`. |
+| SMTP certificate verification | `EMAIL` over `use_ssl` or `use_tls` verifies the server certificate (upstream accepted any certificate, so the SMTP password could go to an impostor). New `[email]` settings: `ca_file` to trust an internal CA, `verify_certificate = No` to turn the check off.                                                                                                                                                                                                            |
 
 ### Bug Fixes
 
@@ -362,6 +362,11 @@ These are behavioral changes driven by security or correctness issues in the ups
 | Partial dates inferred as timestamps                               | `dateutil` fills fields missing from a value with today's date, so `1-2`, `5/6`, `March 5`, `Jan` and `Monday` were inferred as timestamps and, on DuckDB, SQL Server, MySQL, Oracle, Firebird, Access and every ODS/XLSX/JSON/Parquet/Feather import, stored as dates in the current year. Inference now requires a four-digit year (or a month-first `m/d/yy` or `m-d-yy`) and a month and day, which restores upstream's behaviour for these values. `m-d-yy` is accepted in addition to upstream's `m/d/yy`.                                                                                                                                                                                                                                                                                                   |
 | Offset values stored as plain timestamps                           | With `dateutil`, a value with a UTC offset (`2024-01-01 10:00+05:00`) also counted as a plain timestamp, so a column mixing offset and plain values was created as `timestamp` and the driver converted the offset values with the client's time zone: the stored value depended on the machine that ran the import. A value with an offset is again not a plain timestamp, as with upstream's offset-free format list, so a mixed column is text and keeps each value as written. A column where every value has an offset is still `timestamptz`.                                                                                                                                                                                                                                                                |
 | Binary columns in `COPY`                                           | Every value a source cursor returned as bytes was decoded to text (undecodable bytes as `\xNN` escapes) before `COPY` wrote it, and `TO NEW` typed the column as text. Upstream did the same for SQLite BLOBs; with psycopg2 a PostgreSQL `bytea` arrived as a `memoryview` and escaped the decode, so for PostgreSQL this was an execsql2 regression. `COPY` now moves bytes unchanged and creates binary columns. Exports still write the decoded text.                                                                                                                                                                                                                                                                                                                                                          |
+| Backslashes in `!'!var!'!`                                         | Upstream doubled backslashes on Windows only, to protect `re.sub`'s replacement template; execsql2 2.x doubled them on every platform as SQL escaping, which corrupted values on every DBMS but MySQL. A backslash is now doubled only where it is an escape: on a MySQL/MariaDB connection, and in a PostgreSQL escape string written as `E!'!var!'!`. Apostrophes are doubled everywhere.                                                                                                                                                                                                                                                                                                                                                                                                                        |
+| Time columns on Oracle                                             | `IMPORT` / `COPY ... TO NEW` of a column of times stopped with "Oracle DBMS type has no specification for data type time" (a `DT_Time_Oracle` type was registered but inference never produced it). A time column is now created as `VARCHAR2(15)` holding the ISO time text.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+| Capitalized headers on MySQL                                       | MySQL compared the file's headers with the table's lower-cased column names, so `Id` did not match `id`: IMPORT stopped (after `TO NEW` had created the table), and with `IMPORT_COMMON_COLUMNS_ONLY` such columns were skipped. The comparison now ignores case, as on PostgreSQL and the other DBMSs.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
+| `SKIP` on PostgreSQL's COPY path                                   | The PostgreSQL fast path (a quoted CSV file) skipped one line, the header, whatever `SKIP` said, so the skipped lines were loaded as data. It now starts after the `SKIP` lines like every other reader.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
+| Spreadsheet rows on import                                         | XLSX reading stopped at the first row with no truthy value, so a row of `0` / FALSE values ended the data; XLS reading with `SKIP n` stopped *n* rows before the end of the sheet. Both now read to the first row of empty cells.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
 | Short data lines on import                                         | A line with fewer values than the header was padded with NULLs only on PostgreSQL and MySQL. SQLite stopped with "Too few values", DuckDB and the other DBMSs with a driver parameter-count error. With `TO NEW` / `REPLACEMENT`, type inference skipped the missing values, so the column could be created `NOT NULL` and then reject the padded line, on PostgreSQL too. Missing trailing values are now NULL on every DBMS, and inference counts them as NULL.                                                                                                                                                                                                                                                                                                                                                  |
 | `COPY` re-typed text values                                        | Upstream described the destination table by re-reading the source's text values as if they came from a file, so a text column of `1`/`0` became `boolean` and one of dates became `timestamp` with `NEW` / `REPLACEMENT`, and values copied into an existing text column were converted on the way in (`1` stored as `true`). Text from the source database now stays text. Upstream also ran the source query twice, once to describe the table and once to copy it; execsql2 runs it once and holds the rows in a temporary file in between.                                                                                                                                                                                                                                                                     |
 | `REPLACEMENT` of a quoted or schema-qualified table                | The SQLite, SQL Server, Oracle, Firebird and Access adapters quoted the table name a second time before dropping it, so `IMPORT` / `COPY ... TO REPLACEMENT "my tbl"` dropped a table named `"my tbl"` (quotes included) that did not exist and then failed to create the table, and on SQL Server, Oracle and Firebird `REPLACEMENT staging.t` tried to drop one table named `staging.t`. The name is now quoted once.                                                                                                                                                                                                                                                                                                                                                                                            |

{execsql2-2.28.0 → execsql2-2.30.0}/docs/api/index.md

@@ -80,7 +80,7 @@ Each run has its own [file writer](#file-output), so two runs writing files at t
 
 Three things are still shared by every thread in the process, so keep them apart yourself:
 
-- **The working directory.** `CD` calls `os.chdir()`, which moves every thread. Use absolute paths in scripts that run concurrently, or don't use `CD` in them.
+- **The working directory.** `CD` calls `os.chdir()`, which moves every thread while the script runs. Use absolute paths in scripts that run concurrently, or don't use `CD` in them. When a script that ran `CD` finishes, `run()` changes back to the directory it was in before that `CD`, so the caller ends where it started; a run without `CD` leaves the directory as it finds it.
 - **The same output file.** Two runs writing one file at the same time can interleave their lines. Give each run its own output paths.
 - **Interactive prompts.** `PROMPT` and the other GUI metacommands are not meant to be driven from several threads at once.
 

{execsql2-2.28.0 → execsql2-2.30.0}/docs/dev/architecture.md

@@ -56,6 +56,8 @@ flowchart LR
     UTILS["utils/<br/>auth, crypto,<br/>fileio, mail,<br/>regex, strings"]
     PARSER["parser.py<br/>CondParser,<br/>NumericParser"]
     DEBUG["debug/<br/>REPL debugger"]
+    LSP["lsp/<br/>execsql lsp,<br/>editor features"]
+    REF["metacommands/<br/>reference.py<br/>syntax registry"]
 
     CLI --> CONFIG
     CLI --> STATE
@@ -75,35 +77,66 @@ flowchart LR
     EXPORT --> STATE
     GUI --> STATE
     SCRIPT --> PARSER
+    CLI --> LSP
+    LSP --> SCRIPT
+    LSP --> REF
+    LSP --> CLI
 ```
 
 ### Package summary
 
-| Package          | Purpose                                                                                                                                                         |
-| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `cli/`           | Typer app and help rendering, one module per command under `cli/commands/`, shared options, `_run()` orchestration, DSN URL parsing                             |
-| `api.py`         | Public `execsql.run()` Python entry point for notebooks, pipelines, and library use                                                                             |
-| `config.py`      | `ConfigData` (INI merging), `StatObj` (runtime flags), `WriteHooks` (stdout/stderr redirection)                                                                 |
-| `state.py`       | Thread-local runtime store — all shared mutable state lives here, isolated per-thread                                                                           |
-| `script/`        | AST node types, parser, `MetaCommandList`, `SubVarSet`, `BatchLevels`, `ScriptExecSpec`, `set_system_vars()`                                                    |
-| `metacommands/`  | `build_dispatch_table()`, all `x_*` handlers, `build_conditional_table()`, all `xf_*` predicates                                                                |
-| `db/`            | `Database` ABC, `DatabasePool`, 9 adapter modules (postgres, sqlite, duckdb, mysql, sqlserver, oracle, firebird, access, dsn)                                   |
-| `exporters/`     | `ExportRecord`, `ExportMetadata`, `WriteSpec`, 20+ format writers (CSV, JSON, XML, HTML, etc.)                                                                  |
-| `importers/`     | `CsvFile`, `OdsFile`, `XlsFile`, `FeatherFile` — data import backends                                                                                           |
-| `gui/`           | `GuiBackend` ABC, `TkinterBackend`, `TextualBackend`, `ConsoleBackend`                                                                                          |
-| `utils/`         | Shared utilities: file I/O, encryption, mail, regex helpers, string manipulation, timers                                                                        |
-| `parser.py`      | Recursive-descent parsers for conditional (`IF`) and arithmetic (`SET`) expressions                                                                             |
-| `types.py`       | `DataType` subclasses and `DbType` per-DBMS type dialect mappings                                                                                               |
-| `models.py`      | `Column`, `DataTable`, `JsonDatatype`                                                                                                                           |
-| `format.py`      | `execsql format` formatter — opinionated formatter for execsql scripts                                                                                          |
-| `exceptions.py`  | `ExecSqlError` base, `ErrInfo`, `ConfigError`, `DataTypeError`, `DbTypeError`, etc.                                                                             |
-| `plugins.py`     | Entry-point plugin discovery for metacommands, exporters, and importers                                                                                         |
-| `debug/`         | Interactive REPL debugger for stepping through script execution                                                                                                 |
-| `interactive.py` | Engine behind the debug REPL: input typed at a BREAKPOINT runs as the next lines of the paused script via `executor.execute_input`; errors return to the prompt |
-| `data/`          | Bundled package data (`execsql.conf.template` powers `execsql config --init`)                                                                                   |
+| Package                | Purpose                                                                                                                                                                                        |
+| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `cli/`                 | Typer app and help rendering, one module per command under `cli/commands/`, shared options, `_run()` orchestration, DSN URL parsing                                                            |
+| `api.py`               | Public `execsql.run()` Python entry point for notebooks, pipelines, and library use                                                                                                            |
+| `config.py`            | `ConfigData` (INI merging), `StatObj` (runtime flags), `WriteHooks` (stdout/stderr redirection)                                                                                                |
+| `state.py`             | Thread-local runtime store — all shared mutable state lives here, isolated per-thread                                                                                                          |
+| `script/`              | AST node types, parser, `MetaCommandList`, `SubVarSet`, `BatchLevels`, `ScriptExecSpec`, `set_system_vars()`                                                                                   |
+| `metacommands/`        | `build_dispatch_table()`, all `x_*` handlers, `build_conditional_table()`, all `xf_*` predicates                                                                                               |
+| `db/`                  | `Database` ABC, `DatabasePool`, 9 adapter modules (postgres, sqlite, duckdb, mysql, sqlserver, oracle, firebird, access, dsn)                                                                  |
+| `exporters/`           | `ExportRecord`, `ExportMetadata`, `WriteSpec`, 20+ format writers (CSV, JSON, XML, HTML, etc.)                                                                                                 |
+| `importers/`           | `CsvFile`, `OdsFile`, `XlsFile`, `FeatherFile` — data import backends                                                                                                                          |
+| `gui/`                 | `GuiBackend` ABC, `TkinterBackend`, `TextualBackend`, `ConsoleBackend`                                                                                                                         |
+| `utils/`               | Shared utilities: file I/O, encryption, mail, regex helpers, string manipulation, timers                                                                                                       |
+| `parser.py`            | Recursive-descent parsers for conditional (`IF`) and arithmetic (`SET`) expressions                                                                                                            |
+| `types.py`             | `DataType` subclasses and `DbType` per-DBMS type dialect mappings                                                                                                                              |
+| `models.py`            | `Column`, `DataTable`, `JsonDatatype`                                                                                                                                                          |
+| `format.py`            | `execsql format` formatter — opinionated formatter for execsql scripts                                                                                                                         |
+| `exceptions.py`        | `ExecSqlError` base, `ErrInfo`, `ConfigError`, `DataTypeError`, `DbTypeError`, etc.                                                                                                            |
+| `plugins.py`           | Entry-point plugin discovery for metacommands, exporters, and importers                                                                                                                        |
+| `debug/`               | Interactive REPL debugger for stepping through script execution                                                                                                                                |
+| `interactive.py`       | Engine behind the debug REPL: input typed at a BREAKPOINT runs as the next lines of the paused script via `executor.execute_input`; errors return to the prompt                                |
+| `lsp/`                 | `execsql lsp`, the language server editors start (pygls 2): diagnostics, completion, hover, navigation, rename, folding, quick fixes, Format Document. See [Language Server](#language-server) |
+| `manifest.py`          | `RunManifest`: the JSON record `--manifest FILE` writes when a run ends (script, database, statements, files read/written/deleted, connections, outcome)                                       |
+| `config_validation.py` | `execsql config --validate`: reads the same config files as a run and reports every problem with its file and line                                                                             |
+| `data/`                | Bundled package data (`execsql.conf.template` powers `execsql config --init`)                                                                                                                  |
 
 ______________________________________________________________________
 
+## Language Server { #language-server }
+
+`execsql lsp` (`cli/commands/lsp.py` → `lsp/server.py:create_server()`) is a pygls 2 server on stdin/stdout. It never runs a script or connects to a database: it reuses the static tools.
+
+| Module                                     | Feature                                                 | Reuses                                                             |
+| ------------------------------------------ | ------------------------------------------------------- | ------------------------------------------------------------------ |
+| `server.py`                                | handlers, settings, debounced lint, watched files       | `cli.run._tool_config`                                             |
+| `diagnostics.py`                           | lint findings as LSP diagnostics                        | `cli.lint.lint_source`                                             |
+| `document.py`                              | `ScriptIndex`: definitions, references, `INCLUDE` graph | `script.parser`, `cli.lint.variables_defined_by`                   |
+| `cache.py`                                 | each text's `FileFacts`, included files' text           | `document`                                                         |
+| `_util.py`                                 | `span()`, `markdown()`, the metacommand-line regex      | `lsprotocol`                                                       |
+| `completion.py`, `hover.py`                | completion and hover                                    | `metacommands.reference`                                           |
+| `navigation.py`, `rename.py`, `folding.py` | definition, references, outline, links, rename, folding | `ScriptIndex`, the parser                                          |
+| `code_actions.py`                          | quick fixes for P002, P003, P004, V001                  | the dispatch table (each fix is re-validated before it is offered) |
+| `formatting.py`                            | Format Document                                         | `format.format_file` with the `[format]` settings                  |
+
+**Settings per workspace folder.** `ConfigData(start_dir=...)` (through `_tool_config(None, start_dir=folder)`) reads config as a run started in that directory would: its `execsql.conf`, and relative `config_file` chains resolved against it. `load_settings()` reads each folder's `[lint]` / `[format]` settings that way and keeps them by folder; a document uses its innermost folder's settings, a document in no folder those of the workspace root (`base_dir()`, or the server's working directory when there is no root or it is gone). The server never changes its working directory. An unsaved script's relative `INCLUDE` resolves against `base_dir()` too, for both lint (`lint_source(base_dir=...)`) and the index.
+
+**Index lifetime.** A `ScriptIndex` is assembled on every request that needs one (hover, completion, definition, references, rename, links, code actions) from per-file `FileFacts`: what one file defines, calls, includes and refers to, independent of the files it includes. `lsp/cache.py` keeps each text's facts and each included file's text (by path, size and modification time, dropped when the client reports the file changed), so a request reparses only the edited buffer; a file with a `SUB_INI` is reparsed every time, since its INI file can change on its own. `INCLUDE`d files are read too (the editor's unsaved copy when the file is open), up to 200 files, 10 MiB each and 20 MiB together, regular files only; `SUB_INI` files at most 1 MiB, cached by path, size and modification time.
+
+**Linting.** On open, and 200 ms after the last edit (`LINT_DELAY_SECONDS`), lint runs on the document and the result is published with the document's version. A change to a watched `**/execsql.conf` reloads the settings and relints every open document. Saving a script, or a change to a watched `**/*.sql`, relints only the open documents that are that file or `INCLUDE` it directly: lint reads no included file's text, it only checks that each exists. Both go through the same debounce, so the editor's `didSave` and its watcher's report of the same save lint once.
+
+**Trust.** The server reads the files the workspace names, so the VS Code client (`extras/vscode-execsql/extension.js`) starts it only in a trusted workspace, resolves `execsql` through `PATH` (never from the open folder) and runs it from the home directory. Plugins load from the server's own Python environment, not from the workspace.
+
 ## AST Parser and Executor
 
 execsql2 parses scripts into an AST and walks the tree to execute. There is no separate flat-command-list engine; the AST executor is the only path.
@@ -157,6 +190,10 @@ ______________________________________________________________________
 
 Metacommands are lines in SQL scripts prefixed with `-- !x!`. At import time, `metacommands/__init__.py` calls `build_dispatch_table()`, which populates a `MetaCommandList` with all `mcl.add()` registrations (~225 regex patterns). This singleton is stored as `_state.metacommandlist`.
 
+### The syntax registry
+
+`metacommands/reference.py` describes every metacommand a second way: the syntax forms as the reference docs write them, a one-line summary, a category and the docs anchor. The regex dispatch table decides what *runs*; the registry is what people and editors *read*. `execsql list metacommands`, the language server's completion snippets and hover text come from it. `tests/test_reference.py` checks that every registry form is accepted by the dispatch table, that every dispatch keyword has an entry, and that each anchor exists in `docs/reference/metacommands.md`. A new metacommand therefore needs both: an `mcl.add()` registration and a registry entry.
+
 ### How dispatch works
 
 For each `MetaCommandStatement` AST node, the executor calls `_exec_metacommand()` in `script/executor.py`, which delegates to `_state.metacommandlist.eval(cmd_str)`. The dispatcher extracts the leading keyword, narrows ~225 entries to a small candidate set via a keyword index, then tests each candidate's compiled regex against the full command. The matched handler is called with the regex's named groups as keyword arguments plus `metacommandline` (the original unmodified line), and its hit counter is incremented.
@@ -303,19 +340,19 @@ This import pattern (always as `_state`, always accessed inside function/method
 
 ### Key state variables
 
-| Variable          | Type                     | Purpose                                                          |
-| ----------------- | ------------------------ | ---------------------------------------------------------------- |
-| `conf`            | `ConfigData`             | Merged configuration                                             |
-| `subvars`         | `SubVarSet`              | Global substitution variables                                    |
-| `ast_exec_stack`  | `list[ExecFrame]`        | Unified execution stack (scopes + IF/LOOP/BATCH/INCLUDE)         |
-| `ast_scripts`     | `dict[str, ScriptBlock]` | Named scripts from `BEGIN/END SCRIPT` (AST registry)             |
-| `last_command`    | `ScriptCmd \| None`      | Most-recently-executed statement; powers `current_script_line()` |
-| `dbs`             | `DatabasePool`           | Open database connections                                        |
-| `metacommandlist` | `MetaCommandList`        | Metacommand dispatch table                                       |
-| `conditionallist` | `MetaCommandList`        | Conditional predicate dispatch table                             |
-| `counters`        | `CounterVars`            | Auto-incrementing counters                                       |
-| `filewriter`      | `FileWriter`             | This run's background file-writing thread                        |
-| `exec_log`        | `Logger`                 | Execution log                                                    |
+| Variable          | Type                         | Purpose                                                          |
+| ----------------- | ---------------------------- | ---------------------------------------------------------------- |
+| `conf`            | `ConfigData`                 | Merged configuration                                             |
+| `subvars`         | `SubVarSet`                  | Global substitution variables                                    |
+| `ast_exec_stack`  | `list[ExecFrame]`            | Unified execution stack (scopes + IF/LOOP/BATCH/INCLUDE)         |
+| `ast_scripts`     | `dict[str, ScriptBlock]`     | Named scripts from `BEGIN/END SCRIPT` (AST registry)             |
+| `last_command`    | `ExecutingStatement \| None` | Most-recently-executed statement; powers `current_script_line()` |
+| `dbs`             | `DatabasePool`               | Open database connections                                        |
+| `metacommandlist` | `MetaCommandList`            | Metacommand dispatch table                                       |
+| `conditionallist` | `MetaCommandList`            | Conditional predicate dispatch table                             |
+| `counters`        | `CounterVars`                | Auto-incrementing counters                                       |
+| `filewriter`      | `FileWriter`                 | This run's background file-writing thread                        |
+| `exec_log`        | `Logger`                     | Execution log                                                    |
 
 ______________________________________________________________________
 
@@ -328,7 +365,7 @@ ______________________________________________________________________
 1. System-wide (platform-dependent)
 1. User home directory
 1. Script file directory
-1. Current working directory
+1. Current working directory (or `start_dir`, when the caller passes one)
 
 It exposes all recognized options as attributes (`db_type`, `server`, `db`, `db_file`, `username`, `script_encoding`, `output_encoding`, etc.).
 

{execsql2-2.28.0 → execsql2-2.30.0}/docs/getting-started/syntax.md

@@ -300,8 +300,8 @@ The options below belong to `run`: `execsql run -tl script.sql mydb.sqlite`.
 `--version` and `-o`/`--online-help` work on their own (`execsql --version`)
 and anywhere on a `run` command line.
 `format` has its own options, described in the [formatter guide](../guides/formatter.md);
-`lint`'s are in the [lint rules reference](../reference/lint.md); `lsp`'s (`--check`) in
-[Editor Support](../guides/editors.md#install); `config`, `list` and `init` are described
+`lint`'s are in the [lint rules reference](../reference/lint.md); `lsp`'s (`--check`,
+`--log-level`, `--log-file`) in [Editor Support](../guides/editors.md#troubleshooting); `config`, `list` and `init` are described
 [above](#config_command).
 
 ### Connection options
@@ -479,7 +479,7 @@ Valid encoding names can be displayed with `execsql list encodings`. See also [C
     }
     ```
 
-    Files and connections are recorded as each metacommand succeeds, after variable substitution, so paths are the ones used (`out/west.csv`, not `out/!!region!!.csv`); `INCLUDE`d scripts count as files read. When a run fails, `exit_status` is its exit code and `errors` holds the message, its `type`, `source`, `line` and the failing `command`. `variables` lists the names set with `-a` and `--var`; their values are never written, for the same reason the log hides them, and neither are passwords.
+    Files and connections are recorded as each metacommand succeeds, after variable substitution, so paths are the ones used (`out/west.csv`, not `out/!!region!!.csv`); `INCLUDE`d scripts count as files read. When a run fails, `exit_status` is its exit code and `errors` holds the message, its `type`, `source`, `line` and the failing `command`. `variables` lists the names set with `-a` and `--var`, not their values. A password (from `--dsn`, a prompt or the keyring), a `SUB_DECRYPT` result, or a common secret shape (a URL's password, `password=...`, an API token) that ends up in a path, a connection target or an error message is written as `***`. Values given with `-a` and `--var` are kept where a path uses them, as in `out/west.csv`, although the log masks them: keep a secret out of `-a` and `--var` if the manifest is collected.
 
     The path is checked before the run starts: if *FILE* cannot be written (its folder is a file, or not writable), `execsql run` exits 2 before any SQL runs.