execsql2 2.27.0__tar.gz → 2.29.0__tar.gz

{execsql2-2.27.0 → execsql2-2.29.0}/CHANGELOG.md

@@ -13,6 +13,44 @@ ______________________________________________________________________
 
 ______________________________________________________________________
 
+## [2.29.0] - 2026-10-13
+
+### Changed
+
+- **`!'!var!'!` stores backslashes as written** on every DBMS: a value such as `C:\data\new` was stored as `C:\\data\\new` on SQLite, PostgreSQL, DuckDB, SQL Server, Oracle, Firebird and Access. MySQL/MariaDB (which read `\` as an escape) and PostgreSQL escape strings written as `E!'!var!'!` still get backslashes doubled, so both forms store the same value and the `E` workaround is no longer needed. Scripts that removed the doubled backslashes some other way should stop.
+
+### Fixed
+
+- **Oracle IMPORT and COPY with a time column** create it as text (`VARCHAR2(15)`, holding `HH:MM:SS`); `TO NEW` stopped with "Oracle DBMS type has no specification for data type time".
+- **MySQL IMPORT accepts a header with capital letters** (`Id,Name`): it stopped with "columns that are not in table" (leaving the new table behind), and with `IMPORT_COMMON_COLUMNS_ONLY` it loaded those columns as NULL.
+- **PostgreSQL IMPORT honors `SKIP n`** for quoted CSV files, which load through PostgreSQL's COPY: it skipped only the header line, loading the lines before it as data or failing on them.
+- **Spreadsheet IMPORT reads every data row:** an XLSX row whose values are all `0` or FALSE no longer ends the data (it and every row after it were dropped), and an XLS sheet imported with `SKIP n` no longer loses its last *n* rows.
+
+______________________________________________________________________
+
+## [2.28.0] - 2026-10-13
+
+### Added
+
+- **`execsql lsp --check`** reports whether the language server can start (exit 0) or what is missing (exit 1).
+
+### Changed
+
+- **Missing-package messages** (database drivers, spreadsheet, YAML and template exports, `PG_UPSERT`, the formatter and the language server) name the execsql2 extra and show how to add it while keeping the extras already installed.
+
+### Fixed
+
+- **`execsql format` keeps optimizer hints and MySQL executable comments:** a statement holding `/*+ ... */` or `/*! ... */` is left as written in both modes. It was rewritten to `/* + ... */` and the hint moved off its keyword, so MySQL skipped the `/*! */` SQL (for example a `WHERE` or `PARTITION BY`) and Oracle/MySQL ignored the hint; re-check files the formatter or its pre-commit hook rewrote.
+- **`execsql format` no longer ends a final statement with a `;` the script did not have:** a script execsql refused for an unfinished last statement became one that ran it.
+- **`execsql format`:** a statement kept as written after another on the same line no longer starts with a space.
+- **`execsql format` without sqlglot** printed its `pip install` hint without the `[formatter]` extra.
+- **`execsql lint` and the language server read a `SUB_INI` file once** however many lines name it; a script naming a large INI file on many lines took minutes to lint and froze the editor. V002 for a variable a `SUB_INI` defines is reported at the `SUB_INI` line, not line 0.
+- **`execsql lsp` with an incompatible pygls** names the installed pygls version instead of saying the `lsp` extra is missing.
+- **`COPY` keeps binary columns** (SQLite/DuckDB BLOB, PostgreSQL `bytea`) byte for byte, and `COPY ... TO NEW` creates them as binary columns. They were copied as escaped text (`\x89PNG...`), and into PostgreSQL as text.
+- **DuckDB `TO NEW`** is no longer refused because a table of the same name exists in an `ATTACH`ed database; only the current database counts.
+
+______________________________________________________________________
+
 ## [2.27.0] - 2026-10-12
 
 ### Added

{execsql2-2.27.0 → execsql2-2.29.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: execsql2
-Version: 2.27.0
+Version: 2.29.0
 Summary: Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables.
 Project-URL: Homepage, https://execsql2.readthedocs.io
 Project-URL: Repository, https://github.com/geocoug/execsql
@@ -95,8 +95,10 @@ Requires-Dist: openpyxl>=3.1; extra == 'dev'
 Requires-Dist: polars>=1.0; extra == 'dev'
 Requires-Dist: pre-commit>=3.5.0; extra == 'dev'
 Requires-Dist: pygls<3,>=2.1; extra == 'dev'
+Requires-Dist: pytest-asyncio>=1.0; extra == 'dev'
 Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
 Requires-Dist: pytest-lsp>=1.0; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
 Requires-Dist: pyyaml>=6.0; extra == 'dev'
 Requires-Dist: ruff>=0.4; extra == 'dev'
 Requires-Dist: sqlglot<30.22,>=30.21.0; extra == 'dev'
@@ -464,7 +466,7 @@ execsql format --no-sql --in-place scripts/
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.27.0
+    rev: v2.29.0
     hooks:
       - id: execsql-format
       - id: execsql-lint

{execsql2-2.27.0 → execsql2-2.29.0}/README.md

@@ -321,7 +321,7 @@ execsql format --no-sql --in-place scripts/
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.27.0
+    rev: v2.29.0
     hooks:
       - id: execsql-format
       - id: execsql-lint

{execsql2-2.27.0 → execsql2-2.29.0}/docs/about/divergence.md

@@ -286,6 +286,7 @@ The legacy flat command-list engine (`_parse_script_lines` / `runscripts` / `Com
 - **Exception hierarchy** — All custom exceptions inherit from `ExecSqlError`, enabling `except ExecSqlError` to catch any execsql-originated error.
 - **Exception chaining** — All `raise` statements inside `except` blocks preserve the original traceback via `from`.
 - **Database and driver error messages** — End with the error itself (`OperationalError: table t already exists`). Upstream appended "in <file> on line N of execsql", a line of its own source; in the package the file was often a driver's and the line number another file's. Where in execsql the error was raised goes to `execsql.log` as `Raised at: execsql/<module>.py:<line>`.
+- **Missing optional package messages** — Name the execsql2 extra that installs the package and how to add it to a `uv tool` or `pipx` install without dropping the extras already there (`uv tool install "execsql2[...,postgres]"`). Upstream pointed at the package's web page.
 - **Config file errors** — An invalid value in `execsql.conf` stops a run with `Configuration error: <file>: <problem>` (and the line, for an unreadable file) plus a pointer to `execsql config --validate`. Upstream printed `Configuration error on line N of execsql.py`, where N was a line of its own source. A duplicate key, a missing section header or a bare `%` is reported the same way; upstream failed with an uncaught exception. Errors raised by `ConfigData` for a file are `ConfigFileError`, a subclass of `ConfigError`.
 - **ASSERT error type** — `ASSERT` failures now use a dedicated `"assert"` error type that produces `**** Assertion failed.` instead of `**** Error in metacommand.`. This distinguishes intentional script-level checks from actual metacommand errors. Upstream did not have `ASSERT`.
 
@@ -360,10 +361,16 @@ These are behavioral changes driven by security or correctness issues in the ups
 | `DT_Timestamp` claims time-only values                             | `dateutil.parser.parse()` silently fills in today's date for bare time strings like `"13:15:45"`, so `DT_Timestamp` matched before `DT_Time` in the inference order. `parse_datetime()` now rejects time-only strings.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | Partial dates inferred as timestamps                               | `dateutil` fills fields missing from a value with today's date, so `1-2`, `5/6`, `March 5`, `Jan` and `Monday` were inferred as timestamps and, on DuckDB, SQL Server, MySQL, Oracle, Firebird, Access and every ODS/XLSX/JSON/Parquet/Feather import, stored as dates in the current year. Inference now requires a four-digit year (or a month-first `m/d/yy` or `m-d-yy`) and a month and day, which restores upstream's behaviour for these values. `m-d-yy` is accepted in addition to upstream's `m/d/yy`.                                                                                                                                                                                                                                                                                                   |
 | Offset values stored as plain timestamps                           | With `dateutil`, a value with a UTC offset (`2024-01-01 10:00+05:00`) also counted as a plain timestamp, so a column mixing offset and plain values was created as `timestamp` and the driver converted the offset values with the client's time zone: the stored value depended on the machine that ran the import. A value with an offset is again not a plain timestamp, as with upstream's offset-free format list, so a mixed column is text and keeps each value as written. A column where every value has an offset is still `timestamptz`.                                                                                                                                                                                                                                                                |
+| Binary columns in `COPY`                                           | Every value a source cursor returned as bytes was decoded to text (undecodable bytes as `\xNN` escapes) before `COPY` wrote it, and `TO NEW` typed the column as text. Upstream did the same for SQLite BLOBs; with psycopg2 a PostgreSQL `bytea` arrived as a `memoryview` and escaped the decode, so for PostgreSQL this was an execsql2 regression. `COPY` now moves bytes unchanged and creates binary columns. Exports still write the decoded text.                                                                                                                                                                                                                                                                                                                                                          |
+| Backslashes in `!'!var!'!`                                         | Upstream doubled backslashes on Windows only, to protect `re.sub`'s replacement template; execsql2 2.x doubled them on every platform as SQL escaping, which corrupted values on every DBMS but MySQL. A backslash is now doubled only where it is an escape: on a MySQL/MariaDB connection, and in a PostgreSQL escape string written as `E!'!var!'!`. Apostrophes are doubled everywhere.                                                                                                                                                                                                                                                                                                                                                                                                                        |
+| Time columns on Oracle                                             | `IMPORT` / `COPY ... TO NEW` of a column of times stopped with "Oracle DBMS type has no specification for data type time" (a `DT_Time_Oracle` type was registered but inference never produced it). A time column is now created as `VARCHAR2(15)` holding the ISO time text.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+| Capitalized headers on MySQL                                       | MySQL compared the file's headers with the table's lower-cased column names, so `Id` did not match `id`: IMPORT stopped (after `TO NEW` had created the table), and with `IMPORT_COMMON_COLUMNS_ONLY` such columns were skipped. The comparison now ignores case, as on PostgreSQL and the other DBMSs.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
+| `SKIP` on PostgreSQL's COPY path                                   | The PostgreSQL fast path (a quoted CSV file) skipped one line, the header, whatever `SKIP` said, so the skipped lines were loaded as data. It now starts after the `SKIP` lines like every other reader.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
+| Spreadsheet rows on import                                         | XLSX reading stopped at the first row with no truthy value, so a row of `0` / FALSE values ended the data; XLS reading with `SKIP n` stopped *n* rows before the end of the sheet. Both now read to the first row of empty cells.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
 | Short data lines on import                                         | A line with fewer values than the header was padded with NULLs only on PostgreSQL and MySQL. SQLite stopped with "Too few values", DuckDB and the other DBMSs with a driver parameter-count error. With `TO NEW` / `REPLACEMENT`, type inference skipped the missing values, so the column could be created `NOT NULL` and then reject the padded line, on PostgreSQL too. Missing trailing values are now NULL on every DBMS, and inference counts them as NULL.                                                                                                                                                                                                                                                                                                                                                  |
 | `COPY` re-typed text values                                        | Upstream described the destination table by re-reading the source's text values as if they came from a file, so a text column of `1`/`0` became `boolean` and one of dates became `timestamp` with `NEW` / `REPLACEMENT`, and values copied into an existing text column were converted on the way in (`1` stored as `true`). Text from the source database now stays text. Upstream also ran the source query twice, once to describe the table and once to copy it; execsql2 runs it once and holds the rows in a temporary file in between.                                                                                                                                                                                                                                                                     |
 | `REPLACEMENT` of a quoted or schema-qualified table                | The SQLite, SQL Server, Oracle, Firebird and Access adapters quoted the table name a second time before dropping it, so `IMPORT` / `COPY ... TO REPLACEMENT "my tbl"` dropped a table named `"my tbl"` (quotes included) that did not exist and then failed to create the table, and on SQL Server, Oracle and Firebird `REPLACEMENT staging.t` tried to drop one table named `staging.t`. The name is now quoted once.                                                                                                                                                                                                                                                                                                                                                                                            |
-| `TO NEW` onto an existing table                                    | `COPY`'s checks that the source exists and that a `NEW` target does not were dead code: each raised its error inside a `try` that discarded it. `IMPORT ... TO NEW` had no check. Both now stop with `Table <name> already exists` before any DDL (when the database cannot say whether the table exists, the command goes ahead). A name without a schema counts only in the schema the table would be created in (MySQL: the current database). The source check was removed instead of revived: PostgreSQL reads an unquoted `COPY Orders` as `orders`, which the check would have reported missing.                                                                                                                                                                                                            |
+| `TO NEW` onto an existing table                                    | `COPY`'s checks that the source exists and that a `NEW` target does not were dead code: each raised its error inside a `try` that discarded it. `IMPORT ... TO NEW` had no check. Both now stop with `Table <name> already exists` before any DDL (when the database cannot say whether the table exists, the command goes ahead). A name without a schema counts only in the schema the table would be created in (MySQL: the current database), and on DuckDB only in the current database, not an `ATTACH`ed one. The source check was removed instead of revived: PostgreSQL reads an unquoted `COPY Orders` as `orders`, which the check would have reported missing.                                                                                                                                         |
 | No `exit` log record on error or halt                              | The log was closed before the `exit` record was written, so a run that ended with an error, a `HALT` or a cancelled prompt left no `exit` line, and Ctrl-C was recorded as `unknown`. Every run now writes exactly one `exit` record; Ctrl-C is `interrupted`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
 | SIGTERM lost the run's log                                         | Python's default SIGTERM action ends the process without running exit handlers, so the buffered `execsql.log` (every record of the run) was never written, the transaction was not rolled back explicitly and a running `SYSTEM_CMD` kept going. The `execsql` command now turns SIGTERM into a normal exit with status 143 and an `exit` record of type `terminated`. `execsql.run()` does not install a handler.                                                                                                                                                                                                                                                                                                                                                                                                 |
 | `access_use_numeric` ignored                                       | The config option was read but never applied, so MS-Access tables created by `IMPORT` / `COPY` always got `DOUBLE` for decimal data. With `Yes` they get `NUMERIC(precision,scale)` again, as upstream. The setting is read when each table is created, so it applies per run (upstream changed the shared type map once at startup).                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |

{execsql2-2.27.0 → execsql2-2.29.0}/docs/getting-started/syntax.md

@@ -300,8 +300,9 @@ The options below belong to `run`: `execsql run -tl script.sql mydb.sqlite`.
 `--version` and `-o`/`--online-help` work on their own (`execsql --version`)
 and anywhere on a `run` command line.
 `format` has its own options, described in the [formatter guide](../guides/formatter.md);
-`lint`'s are in the [lint rules reference](../reference/lint.md); `config`, `list` and `init`
-are described [above](#config_command).
+`lint`'s are in the [lint rules reference](../reference/lint.md); `lsp`'s (`--check`) in
+[Editor Support](../guides/editors.md#install); `config`, `list` and `init` are described
+[above](#config_command).
 
 ### Connection options
 

{execsql2-2.27.0 → execsql2-2.29.0}/docs/guides/editors.md

@@ -123,9 +123,8 @@ pipx install --force "execsql2[postgres,lsp]"
 ```
 
 See [Adding an extra later](../getting-started/installation.md#adding-an-extra-later).
-Check it with `execsql lsp < /dev/null` (on Windows, `execsql lsp < NUL`): it
-exits at once without output when the server is installed, and says the `lsp`
-extra is needed when it is not.
+Check it with `execsql lsp --check`: it prints `execsql lsp is ready (pygls 2.x.y)` and exits 0 when the server can start, and otherwise says what is
+missing and exits 1.
 
 ## Settings { #settings }
 
@@ -163,7 +162,8 @@ execsql is the metacommand [plugins](../dev/architecture.md#plugin-system) insta
 environment, so lint knows their keywords, as `execsql lint` does. Besides the
 open script and the configuration files, it reads the files the script names
 in `INCLUDE` (to find the variables they define: up to 200 files of at most
-10 MiB each) and in `SUB_INI` (at most 1 MiB). It reads only regular files,
+10 MiB each and 20 MiB together) and in `SUB_INI` (at most 1 MiB, read once
+however many lines name it). It reads only regular files,
 never a FIFO or a device.
 
 ## VS Code { #vscode }

{execsql2-2.27.0 → execsql2-2.29.0}/docs/guides/formatter.md

@@ -130,7 +130,7 @@ Conditions joined by `AND` / `OR` go on one line when their combined text is at
 
 #### Layout only, or rewrite { #layout-only }
 
-sqlglot does more than lay SQL out: it writes the SQL it parsed in its own preferred form. By default the formatter changes only layout and keyword case. A statement sqlglot would also rewrite is kept exactly as written, while the statements around it are still formatted.
+sqlglot does more than lay SQL out: it writes the SQL it parsed in its own preferred form. By default the formatter changes only layout and the case of keywords and unquoted names (a function written `myFunc(x)` becomes `MYFUNC(x)`; quoted names and strings keep their case). A statement sqlglot would also rewrite is kept exactly as written, while the statements around it are still formatted.
 
 | You wrote                   | sqlglot writes                        |
 | --------------------------- | ------------------------------------- |
@@ -149,6 +149,15 @@ rewrite_sql = Yes
 
 With it on, the rewrites a statement gets can change when the pinned sqlglot changes (see above). Turning it off later does not undo rewrites already written; it only stops new ones.
 
+In either mode, a statement holding an optimizer hint (`/*+ ... */`, Oracle and MySQL) or a MySQL executable comment (`/*! ... */`) is kept exactly as written: the server reads them as part of the statement, and sqlglot would turn them into ordinary comments or move a hint away from its keyword.
+
+```sql
+INSERT /*+ APPEND */ INTO sales_hist SELECT * FROM sales_stage;
+SELECT a FROM t /*!50001 WHERE a > 1 */;
+```
+
+The formatter never adds a `;` to a final statement that has none: execsql refuses to run a script that ends with an unfinished statement, and formatting leaves that for you to fix.
+
 #### Comment handling
 
 Comments interleaved within SQL statements (e.g. `--` comments between SELECT columns, or inside CASE expressions) are preserved through formatting using a marker-based round-trip:
@@ -261,7 +270,7 @@ select id,name,created_at from users where active = true order by name;
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.27.0
+    rev: v2.29.0
     hooks:
       - id: execsql-format
       - id: execsql-lint # optional: fail the commit on lint errors

{execsql2-2.27.0 → execsql2-2.29.0}/docs/reference/lint.md

@@ -158,7 +158,7 @@ The `execsql-lint` [pre-commit](https://pre-commit.com/) hook runs `execsql lint
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.27.0
+    rev: v2.29.0
     hooks:
       - id: execsql-lint
         args: [--ignore, V002] # optional: any execsql lint option

{execsql2-2.27.0 → execsql2-2.29.0}/docs/reference/metacommands.md

@@ -1950,6 +1950,8 @@ The delimiter characters that will be recognized in a text file, and that can be
 
 The SKIP key phrase specifies the number of lines (or rows) at the beginning of the file (or worksheet) to discard before evaluating the remainder of the input as a data table.
 
+A worksheet's data ends at the first row whose cells are all empty; a row of zeros or FALSE values is data.
+
 If the NEW keyword is used, the input will be scanned to determine the data type of each column, and a CREATE TABLE statement run to create a new table for the data. A column is typed as a date or timestamp only if every value gives a full date with the year written out (`2024-01-05`, `1/5/2024`, `Jan 5, 2024`) or as a month-first two-digit year (`1/5/24`, `01-05-24`); values such as `1-2`, `5/6`, `March 5`, `Jan`, `Monday` or a day-first `13-05-24` leave the column as text. Scanning of the file to determine data formats is separate from the scanning that may be done to determine the quote and delimiter characters. If a table of that name already exists, the IMPORT stops before creating or loading anything, with the error `Table <name> already exists`. A table name without a schema is looked for only in the schema the table would be created in (on MySQL, the current database), so a table of the same name in another schema does not stop the IMPORT. If the table already exists when the NEW keyword is used, a fatal error will result and *execsql* will halt. If the REPLACEMENT keyword is used, the result is the same as if the NEW keyword were used, except that an existing table of the given name will be deleted first. This replacement is not atomic: if the drop succeeds but a later create or load step fails, the old table has already been removed. If the drop fails, *execsql* stops the IMPORT command before creating or loading the replacement table. *execsql* uses a "drop table" statement to drop an existing table, and the "drop table" statement may not succeed if there are dependencies on that table (see the discussion of [implicit drop table
 statements](../guides/sql_syntax.md#implicit_drop)). If the table to be dropped does not exist, an informational message will be written to the log.
 

{execsql2-2.27.0 → execsql2-2.29.0}/docs/reference/security.md

@@ -184,7 +184,7 @@ SELECT * FROM !!target_schema!!.mytable;
 
 If a variable value is derived from external input (command-line arguments, environment variables, `PROMPT` responses, or data read from a file), the script author is responsible for quoting or validating that value before it is substituted into SQL.
 
-The `!'!varname!'!` dereferencing form wraps the value in single quotes and doubles any embedded apostrophes, which helps when inserting text values into SQL literals:
+The `!'!varname!'!` dereferencing form wraps the value in single quotes and doubles any embedded apostrophes (and, on MySQL/MariaDB, backslashes, so `\'` cannot end the literal there), which helps when inserting text values into SQL literals:
 
 ```sql
 -- !x! sub author_name !!&AUTHOR!!

{execsql2-2.27.0 → execsql2-2.29.0}/docs/reference/substitution_vars.md

@@ -11,7 +11,7 @@ SUB <match_string> <replacement_string>
 The \<match_string> identifies the substitution variable; the \<replacement_string> is the text inserted in its place. A substitution is recognized only when the match string is bracketed by one of three tokens:
 
 - `!!name!!` — insert the value verbatim.
-- `!'!name!'!` — wrap the value in single quotes, doubling any embedded apostrophe. Suitable for SQL string literals containing user data.
+- `!'!name!'!` — wrap the value in single quotes, doubling any embedded apostrophe; on a MySQL or MariaDB connection, which reads `\` in a string literal as an escape, backslashes are doubled too. The value is stored exactly as written (`C:\data\new` stays `C:\data\new`). Suitable for SQL string literals containing user data. On PostgreSQL, `E!'!name!'!` (an escape string) doubles backslashes too, so it stores the same value as `!'!name!'!`; the `E` is not needed.
 - `!"!name!"!` — wrap the value in double quotes. Suitable for SQL quoted identifiers, or for metacommand arguments that must be quoted.
 
 Variables can also be set before the script starts: with `--var NAME=VALUE` on the [command line](../getting-started/syntax.md#options), or in the `[variables]` section of a [config file](configuration.md#configuration). `--var` wins over the config file, and a `SUB` in the script can reassign either.

{execsql2-2.27.0 → execsql2-2.29.0}/extras/vscode-execsql/README.md

@@ -196,9 +196,8 @@ ______________________________________________________________________
 
 **No lint findings or completion**
 
-- Run `execsql lsp < /dev/null` in a terminal (on Windows: `execsql lsp < NUL`). It exits at once
-    without output when the server is installed; otherwise it says the `lsp` extra is needed. Add the
-    extra with the extras you already have, e.g. `uv tool install "execsql2[postgres,lsp]"`
+- Run `execsql lsp --check` in a terminal. It says `execsql lsp is ready` when the server can start;
+    otherwise it says what is missing. Add the extra with the extras you already have, e.g. `uv tool install "execsql2[postgres,lsp]"`
     ([Adding an extra later](https://execsql2.readthedocs.io/en/latest/getting-started/installation/#adding-an-extra-later)).
 - If `execsql` works in a terminal but not in VS Code, set `execsql.server.path` to its full path
     (`which execsql`).

{execsql2-2.27.0 → execsql2-2.29.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "execsql2"
-version = "2.27.0"
+version = "2.29.0"
 description = "Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables."
 readme = { file = "README.md", content-type = "text/markdown" }
 license = { file = "LICENSE.txt" }
@@ -103,6 +103,8 @@ dev = [
     "mkdocstrings-python>=2.0.3",
     "mypy>=1.10",
     "pre-commit>=3.5.0",
+    "pytest>=8",
+    "pytest-asyncio>=1.0",
     "pytest-cov>=5.0.0",
     "pytest-lsp>=1.0",
     "ruff>=0.4",
@@ -187,7 +189,7 @@ line-ending = "auto"
 exclude = ["*.md"]
 
 [tool.bumpversion]
-current_version = "2.27.0"
+current_version = "2.29.0"
 commit = true
 tag = true
 tag_name = "v{new_version}"

{execsql2-2.27.0 → execsql2-2.29.0}/src/execsql/cli/commands/format.py

@@ -5,6 +5,7 @@ from __future__ import annotations
 from pathlib import Path
 
 import typer
+from rich.markup import escape
 
 from execsql.cli.application import ExecsqlCommand, app
 from execsql.cli.help import _err_console
@@ -90,7 +91,7 @@ def format_cmd(
         try:
             _require_sqlglot()
         except ImportError as exc:
-            _err_console.print(f"[bold red]Error:[/bold red] {exc}", highlight=False)
+            _err_console.print(f"[bold red]Error:[/bold red] {escape(str(exc))}", highlight=False)
             raise typer.Exit(code=1) from exc
     raise typer.Exit(
         code=run_formatter(

execsql2-2.29.0/src/execsql/cli/commands/lsp.py

@@ -0,0 +1,64 @@
+"""The ``lsp`` command: the language server editors start for ``.sql`` files."""
+
+from __future__ import annotations
+
+import typer
+from rich.markup import escape
+
+from execsql.cli.application import ExecsqlCommand, app
+from execsql.cli.help import _err_console
+from execsql.utils.errors import install_hint
+
+__all__ = ["lsp_cmd"]
+
+
+def _lsp_libraries_installed() -> bool:
+    """Whether pygls and lsprotocol are installed (whether or not this execsql can use them)."""
+    from importlib.util import find_spec
+
+    return all(find_spec(name) is not None for name in ("pygls", "lsprotocol"))
+
+
+def _pygls_version() -> str:
+    from importlib.metadata import PackageNotFoundError, version
+
+    try:
+        return version("pygls")
+    except PackageNotFoundError:
+        return "unknown"
+
+
+@app.command(
+    cls=ExecsqlCommand,
+    name="lsp",
+    help=(
+        "Run the execsql language server on stdin/stdout, for an editor to start. "
+        "Shows execsql lint findings as you type, completes metacommands, "
+        "conditional tests and variables, explains them on hover, and finds where variables, "
+        "SCRIPTs and included files are defined, offers quick fixes, and formats documents. Requires the lsp "
+        "extra; --check reports whether it is installed."
+    ),
+)
+def lsp_cmd(
+    # Editors' language clients pass --stdio (VS Code always does); stdio is the only transport.
+    stdio: bool = typer.Option(False, "--stdio", hidden=True),
+    check: bool = typer.Option(False, "--check", help="Report whether the language server can start, and exit."),
+) -> None:
+    """The ``lsp`` command; its user-facing help is the decorator's ``help``."""
+    try:
+        from execsql.lsp.server import create_server
+    except ImportError as exc:
+        if _lsp_libraries_installed():
+            # pygls is there but this execsql cannot use it (another major version, a broken install).
+            message = (
+                f"execsql lsp cannot use the installed pygls {_pygls_version()} ({exc}). "
+                f"It needs pygls 2.x, which the lsp extra installs. {install_hint('lsp')}"
+            )
+        else:
+            message = f"execsql lsp needs the lsp extra. {install_hint('lsp')}"
+        _err_console.print(f"[bold red]Error:[/bold red] {escape(message)}", highlight=False)
+        raise typer.Exit(code=1) from exc
+    if check:
+        typer.echo(f"execsql lsp is ready (pygls {_pygls_version()}).")
+        return
+    create_server().start_io()

{execsql2-2.27.0 → execsql2-2.29.0}/src/execsql/cli/lint.py

@@ -48,6 +48,7 @@ Public surface:
 
 from __future__ import annotations
 
+import functools
 import json
 import re
 from collections import Counter
@@ -460,23 +461,40 @@ def _read_ini_vars(
     language server; a file that is unreadable or not valid INI defines
     nothing.
     """
-    from configparser import ConfigParser
-    from configparser import Error as ConfigError
-
     p = Path(ini_file)
     if not p.is_absolute() and script_dir is not None:
         p = script_dir / p
+    try:
+        if not p.is_file():
+            return
+        stat = p.stat()
+        if stat.st_size > _MAX_INI_BYTES:
+            return
+        resolved = str(p.resolve())
+    except OSError:
+        return
+    defined_vars.update(_ini_section_keys(resolved, stat.st_size, stat.st_mtime_ns, section))
+
+
+@functools.lru_cache(maxsize=64)
+def _ini_section_keys(path: str, size: int, mtime_ns: int, section: str) -> frozenset[str]:
+    """The upper-cased keys of *section* in the INI file at *path*.
+
+    Cached by the file's size and modification time: a script that names
+    the same file on many SUB_INI lines, and the language server re-linting
+    on every edit, read it once.
+    """
+    from configparser import ConfigParser
+    from configparser import Error as ConfigError
 
     cp = ConfigParser()
     try:
-        if not p.is_file() or p.stat().st_size > _MAX_INI_BYTES:
-            return
-        cp.read_string(p.read_text(encoding="utf-8"), source=str(p))
+        cp.read_string(Path(path).read_text(encoding="utf-8"), source=path)
     except (OSError, UnicodeDecodeError, ConfigError):
-        return
-    if cp.has_section(section):
-        for key, _value in cp.items(section):
-            defined_vars.add(key.upper())
+        return frozenset()
+    if not cp.has_section(section):
+        return frozenset()
+    return frozenset(key.upper() for key, _value in cp.items(section))
 
 
 def _is_known_var(raw_name: str, defined_vars: set[str]) -> bool:
@@ -834,7 +852,12 @@ def _referencing_text(node: Node) -> list[str]:
     return texts
 
 
-def _check_unused_variables(script: Script, defined: set[str], issues: list[_Issue]) -> None:
+def _check_unused_variables(
+    script: Script,
+    defined: set[str],
+    issues: list[_Issue],
+    script_dir: Path | None = None,
+) -> None:
     """Report variables a SUB defines that nothing ever reads.
 
     Almost always a typo in either the definition or the reference — the two
@@ -845,8 +868,11 @@ def _check_unused_variables(script: Script, defined: set[str], issues: list[_Iss
     referenced: set[str] = set()
     _collect_var_references(script.body, referenced)
     unused = sorted(defined - referenced)
+    lines: dict[str, int] = {}
+    if unused:
+        _definition_lines(script.body, script_dir, lines)
     for name in unused:
-        line = _definition_line(script.body, name)
+        line = lines.get(name, 0)
         issues.append(
             _issue(
                 "V002",
@@ -857,18 +883,15 @@ def _check_unused_variables(script: Script, defined: set[str], issues: list[_Iss
         )
 
 
-def _definition_line(nodes: list[Node], name: str) -> int:
-    """Line of the SUB that defines *name*, or 0 when it cannot be located."""
+def _definition_lines(nodes: list[Node], script_dir: Path | None, lines: dict[str, int]) -> None:
+    """Add to *lines* the line of the first metacommand that defines each variable, in script order."""
     for node in nodes:
         if isinstance(node, MetaCommandStatement):
             probe: set[str] = set()
-            _extract_var_definition(node.command, None, probe)
-            if name in probe:
-                return node.span.start_line
-        found = _definition_line(list(node.children()), name)
-        if found:
-            return found
-    return 0
+            _extract_var_definition(node.command, script_dir, probe)
+            for name in probe:
+                lines.setdefault(name, node.span.start_line)
+        _definition_lines(list(node.children()), script_dir, lines)
 
 
 # ---------------------------------------------------------------------------
@@ -1008,7 +1031,7 @@ def lint(
     )
 
     # Pass 3: whole-script rules, which need every reference collected first
-    _check_unused_variables(script, all_defined, issues)
+    _check_unused_variables(script, all_defined, issues, script_dir)
 
     return issues
 

{execsql2-2.27.0 → execsql2-2.29.0}/src/execsql/db/access.py

@@ -17,7 +17,7 @@ from typing import Any, cast
 from execsql.db.base import Database, _fetched
 from execsql.db.tiers import SupportTier
 from execsql.exceptions import ErrInfo
-from execsql.utils.errors import exception_desc, fatal_error
+from execsql.utils.errors import exception_desc, fatal_error, install_hint
 from execsql.utils.auth import (
     begin_login,
     clear_stored_password,
@@ -68,7 +68,7 @@ class AccessDatabase(Database):
         try:
             import pyodbc  # noqa: F401
         except Exception:
-            fatal_error("The pyodbc module is required.  See http://github.com/mkleehammer/pyodbc")
+            fatal_error(f"The pyodbc module is required to connect to Access. {install_hint('odbc')}")
         from execsql.types import dbt_access
 
         super().__init__(
@@ -327,8 +327,8 @@ class AccessDatabase(Database):
             rows = curs.fetchall()
             return [d[0] for d in curs.description], rows
 
-    def select_rowsource(self, sql: str) -> tuple[list[str], Any]:
-        """Return column names and an iterable that yields rows one at a time."""
+    def select_rowsource(self, sql: str, *, decode_bytes: bool = True) -> tuple[list[str], Any]:  # noqa: ARG002
+        """Return column names and an iterable that yields rows one at a time (values as the driver returns them)."""
         # Return 1) a list of column names, and 2) an iterable that yields rows.
         self.dao_flush_check()
         curs = self.cursor()

{execsql2-2.27.0 → execsql2-2.29.0}/src/execsql/db/base.py

@@ -215,21 +215,18 @@ class Database(ABC):
     def quote_literal(self, value: Any) -> str:
         """Return *value* as a SQL string literal, safely escaped.
 
-        Default ANSI/Postgres behaviour: wrap in single quotes, double
-        embedded apostrophes, escape backslashes (so a value containing
-        ``\\'`` cannot terminate the literal on MySQL default mode or
-        PostgreSQL E-strings), and reject embedded NUL bytes (most
-        wire protocols truncate or reject them).
-
-        Override per-DBMS only when the wire protocol requires a different
-        literal form.
+        Standard SQL: wrap in single quotes and double embedded apostrophes;
+        a backslash is an ordinary character.  Embedded NUL bytes are
+        rejected (most wire protocols truncate or reject them).  A DBMS
+        that reads a backslash in a literal as an escape (MySQL) overrides
+        this.
         """
         if value is None:
             return "NULL"
         s = str(value)
         if "\x00" in s:
             raise ValueError("SQL literal contains a NUL byte; refusing to quote.")
-        return "'" + s.replace("\\", "\\\\").replace("'", "''") + "'"
+        return "'" + s.replace("'", "''") + "'"
 
     def paramsubs(self, paramcount: int) -> str:
         """Return a comma-separated string of *paramcount* parameter placeholders."""
@@ -405,8 +402,13 @@ class Database(ABC):
             rows = curs.fetchall()
             return [d[0] for d in curs.description], rows
 
-    def select_rowsource(self, sql: str) -> tuple[list[str], Generator]:
-        """Execute *sql* and return ``(column_names, row_generator)`` for streaming large result sets."""
+    def select_rowsource(self, sql: str, *, decode_bytes: bool = True) -> tuple[list[str], Generator]:
+        """Execute *sql* and return ``(column_names, row_generator)`` for streaming large result sets.
+
+        A binary value (``bytes``) is decoded to text with the connection's
+        encoding, as the exporters expect, unless *decode_bytes* is false:
+        COPY passes false so binary columns reach the target unchanged.
+        """
         # Return 1) a list of column names, and 2) an iterable that yields rows.
         curs = self.cursor()
         try:
@@ -433,7 +435,7 @@ class Database(ABC):
                         break
                     else:
                         for row in rows:
-                            if self.encoding:
+                            if self.encoding and decode_bytes:
                                 yield [
                                     c.decode(self.encoding, "backslashreplace") if isinstance(c, bytes) else c
                                     for c in row
@@ -501,6 +503,15 @@ class Database(ABC):
         """The schema (MySQL: database) an unqualified table name is created in, or ``None`` if unknown."""
         return None
 
+    def table_exists_where_created(self, table_name: str, schema_name: str | None = None) -> bool:
+        """Whether *table_name* exists where a ``CREATE TABLE`` on this connection would put it.
+
+        The same as :meth:`table_exists`, except on a connection that sees
+        tables of other databases (DuckDB ``ATTACH``): only the current
+        database counts there.
+        """
+        return self.table_exists(table_name, schema_name)
+
     def _query_current_schema(self, sql: str) -> str | None:
         with self._cursor() as curs:
             curs.execute(sql)

{execsql2-2.27.0 → execsql2-2.29.0}/src/execsql/db/dsn.py

@@ -12,7 +12,7 @@ the CLI.
 from execsql.db.base import Database
 from execsql.db.tiers import SupportTier
 from execsql.exceptions import ErrInfo
-from execsql.utils.errors import exception_desc, fatal_error
+from execsql.utils.errors import exception_desc, fatal_error, install_hint
 from execsql.utils.auth import (
     begin_login,
     clear_stored_password,
@@ -51,7 +51,7 @@ class DsnDatabase(Database):
         try:
             import pyodbc  # noqa: F401
         except Exception:
-            fatal_error("The pyodbc module is required.  See http://github.com/mkleehammer/pyodbc")
+            fatal_error(f"The pyodbc module is required to connect through an ODBC DSN. {install_hint('odbc')}")
         from execsql.types import dbt_dsn
 
         super().__init__(