execsql2 2.31.2__tar.gz → 2.33.0__tar.gz

{execsql2-2.31.2 → execsql2-2.33.0}/CHANGELOG.md

@@ -13,6 +13,33 @@ ______________________________________________________________________
 
 ______________________________________________________________________
 
+## [2.33.0] - 2026-10-16
+
+### Added
+
+- **`$PG_UPSERT_STATUS`** says how a `PG_UPSERT` ended (`committed`, `dry_run`, `no_changes`, `qa_failed`, `cancelled`, `db_error`), and **`$PG_UPSERT_ERROR`** holds the database error message. A run with nothing to load and a cancelled run used to look the same in the other variables.
+
+### Fixed
+
+- **A failed metacommand sets `$LAST_ERROR` and `$ERROR_MESSAGE`** with `METACOMMAND_ERROR_HALT OFF`, so a script can tell what failed, not only that `METACOMMAND_ERROR()` is true; an `ON ERROR_HALT` handler (including `ON ERROR_HALT WRITE "…!{$LAST_ERROR}!"`) now gets `$LAST_ERROR` too. The error is also written to `execsql.log`.
+- **`execsql.run()` reports a metacommand that failed under `METACOMMAND_ERROR_HALT OFF`** in `result.errors`, with `success` False, as it already did for a SQL error under `ERROR_HALT OFF`.
+- **`PG_UPSERT` cancelled from a prompt** no longer leaves an earlier run's `$PG_UPSERT_*` values (such as `$PG_UPSERT_COMMITTED` `TRUE`) in place.
+
+______________________________________________________________________
+
+## [2.32.0] - 2026-10-16
+
+### Changed
+
+- **Working out column types is about twice as fast** for `IMPORT`, `COPY` and the exports that create a table (`EXPORT … AS SQLITE`, `DUCKDB`, `JSON`, `FEATHER`); a CSV whose dates repeat scans close to three times as fast. The types chosen are the same.
+- **`IMPORT`, `COPY` and `EXPORT … AS DUCKDB` into DuckDB are about five times as fast**: they insert many rows per statement. Copying 200,000 rows took 124 s and now takes 22 s.
+
+### Fixed
+
+- **A column of large integers**, such as epoch milliseconds, no longer makes `IMPORT`, `COPY` or those exports hang or run out of memory while execsql works out the column types.
+
+______________________________________________________________________
+
 ## [2.31.2] - 2026-10-16
 
 ### Fixed

{execsql2-2.31.2 → execsql2-2.33.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: execsql2
-Version: 2.31.2
+Version: 2.33.0
 Summary: Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables.
 Project-URL: Homepage, https://execsql2.readthedocs.io/en/stable/
 Project-URL: Repository, https://github.com/geocoug/execsql
@@ -441,7 +441,7 @@ execsql format --no-sql --in-place scripts/
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.31.2
+    rev: v2.33.0
     hooks:
       - id: execsql-format
       - id: execsql-lint

{execsql2-2.31.2 → execsql2-2.33.0}/README.md

@@ -321,7 +321,7 @@ execsql format --no-sql --in-place scripts/
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.31.2
+    rev: v2.33.0
     hooks:
       - id: execsql-format
       - id: execsql-lint

{execsql2-2.31.2 → execsql2-2.33.0}/docs/about/divergence.md

@@ -48,15 +48,15 @@ ______________________________________________________________________
 
 ### Metacommands
 
-| Metacommand             | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
-| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `ASSERT`                | Evaluate a condition and raise an error (halting the script) if it is false. Supports all IF conditions. Optional quoted failure message. Skipped in false IF blocks.                                                                                                                                                                                                                                                                                          |
-| `BREAKPOINT`            | Pause script execution and drop into an interactive debug REPL. See [Debugging](#debugging) below for full details.                                                                                                                                                                                                                                                                                                                                            |
-| `CONFIG SHOW_PROGRESS`  | Enable the Rich progress bar for IMPORT operations at runtime.                                                                                                                                                                                                                                                                                                                                                                                                 |
-| `CONFIG LOG_SQL`        | Enable SQL query audit logging — writes executed SQL to the log file.                                                                                                                                                                                                                                                                                                                                                                                          |
-| `PG_UPSERT`             | QA-checked, FK-dependency-ordered upserts from staging to base schema on PostgreSQL via the optional [pg-upsert](https://pg-upsert.readthedocs.io/) dependency. Modes: `PG_UPSERT` (full pipeline), `PG_UPSERT QA` (checks only), `PG_UPSERT CHECK` (schema only). QA checks only the rows the given `METHOD` would write. QA failure does not raise — the result is reported via `$PG_UPSERT_QA_PASSED` so the caller's `IF` / `ASSERT` decides control flow. |
-| `IMPORT … FROM JSON`    | Import a JSON file (array of objects or [JSON Lines](https://jsonlines.org/)) into a database table. Nested objects are flattened with dot-separated column names; arrays are stored as JSON strings.                                                                                                                                                                                                                                                          |
-| `SHOW SCRIPTS [<name>]` | Without a name, lists all registered SCRIPT definitions with parameter signatures and source locations. With a name, shows detail including parameters (with defaults), source file/line range, and docstring.                                                                                                                                                                                                                                                 |
+| Metacommand             | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
+| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `ASSERT`                | Evaluate a condition and raise an error (halting the script) if it is false. Supports all IF conditions. Optional quoted failure message. Skipped in false IF blocks.                                                                                                                                                                                                                                                                                                                                      |
+| `BREAKPOINT`            | Pause script execution and drop into an interactive debug REPL. See [Debugging](#debugging) below for full details.                                                                                                                                                                                                                                                                                                                                                                                        |
+| `CONFIG SHOW_PROGRESS`  | Enable the Rich progress bar for IMPORT operations at runtime.                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| `CONFIG LOG_SQL`        | Enable SQL query audit logging — writes executed SQL to the log file.                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+| `PG_UPSERT`             | QA-checked, FK-dependency-ordered upserts from staging to base schema on PostgreSQL via the optional [pg-upsert](https://pg-upsert.readthedocs.io/) dependency. Modes: `PG_UPSERT` (full pipeline), `PG_UPSERT QA` (checks only), `PG_UPSERT CHECK` (schema only). QA checks only the rows the given `METHOD` would write. QA failure does not raise — the result is reported via `$PG_UPSERT_QA_PASSED` so the caller's `IF` / `ASSERT` decides control flow; `$PG_UPSERT_STATUS` says how the run ended. |
+| `IMPORT … FROM JSON`    | Import a JSON file (array of objects or [JSON Lines](https://jsonlines.org/)) into a database table. Nested objects are flattened with dot-separated column names; arrays are stored as JSON strings.                                                                                                                                                                                                                                                                                                      |
+| `SHOW SCRIPTS [<name>]` | Without a name, lists all registered SCRIPT definitions with parameter signatures and source locations. With a name, shows detail including parameters (with defaults), source file/line range, and docstring.                                                                                                                                                                                                                                                                                             |
 
 ### SCRIPT Enhancements
 
@@ -264,6 +264,7 @@ All 33 mutable runtime globals in `state.py` have been consolidated into a `Runt
 - **PostgreSQL driver is psycopg3** — the PostgreSQL adapter uses `psycopg` (psycopg3, installed via the `psycopg[binary]` extra) instead of upstream's `psycopg2`. The connection now passes the libpq `dbname` keyword (psycopg3 rejects psycopg2's `database`), `VACUUM` toggles `conn.autocommit` instead of the removed `set_session()`, server-side `COPY` import uses psycopg3's `cursor.copy()` context manager instead of `copy_expert()`, and binary `IMPORT` sends raw `bytes` instead of `psycopg2.Binary()`. This also makes `db.conn` directly usable by the `PG_UPSERT` metacommand's pg-upsert ≥1.26.0 dependency, which itself requires a psycopg3 connection. The connection sets `prepare_threshold=None` to disable psycopg3's automatic server-side prepared statements, which psycopg2 never used; without this, re-running a query (e.g. via `EXPORT`) against an object the script drops/recreates or alters raises `FeatureNotSupported: cached plan must not change result type`.
 - **Oracle driver is python-oracledb** — the Oracle adapter imports `oracledb`, the driver the `oracle` extra installs, instead of upstream's `cx_Oracle` (which ships no wheels for Python 3.12+). The `makedsn` / `connect` calls are the same. `cx_Oracle` is still used when it is the only Oracle driver installed.
 - **SQL Server tries ODBC Driver 18 first** — upstream's driver list stopped at Driver 17, so a machine with only Microsoft's current driver could not connect. Driver 18 encrypts the connection and verifies the server's certificate; the `trust_server_certificate` setting accepts a self-signed one.
+- **DuckDB bulk inserts** — `IMPORT`, `COPY` and `EXPORT ... AS DUCKDB` insert many rows per `INSERT ... VALUES` statement instead of `executemany()`, which DuckDB runs one row at a time; the values stored are the same. While binding them, and only when pandas is not installed, `sys.modules["pandas"]` is `None`: the duckdb package looks pandas up twice per row, and without that entry each lookup searched `sys.path`.
 - **Each pyodbc connection is a new session** — the SQL Server, Access and ODBC DSN adapters set `pyodbc.pooling = False` before they connect. With pyodbc's default pooling, unixODBC (macOS, Linux) handed a closed connection's session unchanged to the next connection to the same server, so temp tables and `SET` options from one `CONNECT`, or one `execsql.run()`, were still there in the next. pyodbc applies the setting at the process's first pyodbc connection, so it also covers other pyodbc connections the process makes, and has no effect if the program connected with pyodbc before execsql did.
 - **`Database` is an ABC** — `open_db()` and `exec_cmd()` are abstract methods. Subclasses that omit them raise `TypeError` at instantiation instead of at call time.
 - **Connection timeouts** — PostgreSQL and SQLite adapters accept a connection timeout parameter (default 30 seconds).
@@ -348,7 +349,7 @@ These are behavioral changes driven by security or correctness issues in the ups
 | `WRITE` after `CD`                                                 | A relative `WRITE ... TO` path is resolved against the working directory at the time of the `WRITE`, so it follows an earlier `CD` the way `EXPORT` and `IMPORT` already did. Upstream wrote it relative to the directory execsql started in, because its writer subprocess never saw the `CD`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
 | Substitution variable cycles                                       | 100-iteration limit prevents infinite loops on cyclic variable references.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
 | Script location in error messages                                  | `ErrInfo.script_file` and `script_line_no` are now populated via `stamp_errinfo()` so error output includes "Line N of script foo.sql" context — restoring behavior present in the monolith.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
-| `$ERROR_MESSAGE` not updated                                       | `$ERROR_MESSAGE` is now set on every error path: `exit_now()`, non-halting SQL errors, and non-halting metacommand errors. Previously it was initialized to `""` and never changed.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
+| `$ERROR_MESSAGE` not updated                                       | `$ERROR_MESSAGE` is now set on every error path: `exit_now()`, non-halting SQL errors, and non-halting metacommand errors. Previously it was initialized to `""` and never changed. `$LAST_ERROR` also holds a failed metacommand, with `METACOMMAND_ERROR_HALT OFF` and in an `ON ERROR_HALT` handler, as the documentation describes; upstream set it only for SQL errors and unknown metacommands.                                                                                                                                                                                                                                                                                                                                                                                                              |
 | Metacommand error message lost                                     | When `halt_on_metacommand_err` is `ON`, the original handler `ErrInfo` is now re-raised; the generic "Unknown metacommand" message no longer replaces the specific error from the handler.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
 | Empty script name in error msg                                     | `_execute_script_direct()` and `_execute_script_textual_console()` no longer append "in script , line 0" to uncaught-exception messages when `current_script_line()` returns an empty string.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
 | `PROMPT COMPARE` diff comparison                                   | Diff engine uses native Python equality instead of string comparison — numeric types, Decimals, and booleans compare correctly. `None` is distinguished from empty string. Columns are matched by name (not position), key columns are excluded from comparison, and duplicate PKs keep the first row.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
@@ -434,6 +435,7 @@ These are behavioral changes driven by security or correctness issues in the ups
 | ODS export replaced newlines inside a value with spaces            | String cells were written after `item.replace("\n", " ")`, so an exported value silently differed from the one in the database. An XML parser does normalise a newline inside an attribute, so `office:string-value` cannot carry one — but ODF represents a multi-line cell as one `<text:p>` per line, which is what LibreOffice writes and what execsql2 now writes. Single-line values still use the attribute.                                                                                                                                                                                                                                                                                                                                                                                                |
 | XLSX export wrote `numeric` columns as text                        | `_cell_value()` passed `int` and `float` straight to openpyxl but let every other type fall through to `str()`. PostgreSQL, MySQL, and DuckDB all return `Decimal` for `numeric`/`decimal` columns, so measurement values arrived in Excel as text cells — invisible in a screenshot, useless in a formula, a sort, or a chart. openpyxl lists `Decimal` in its own `NUMERIC_TYPES` and stores it as a real numeric cell, so the conversion was never needed. Found by exporting from a live PostgreSQL container and checking `cell.data_type` rather than `cell.value`.                                                                                                                                                                                                                                          |
 | `EMAIL` SMTP login without a password                              | Upstream calls `smtplib.login()` with only a username when `[email] password` and `enc_password` are both unset, which raises `TypeError` from the stdlib — `login()` has no single-argument form. execsql2 reports a configuration error naming the missing password instead.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
+| Type scan of large integers                                        | Working out a column's type tested each integer against the binary type by building `bytearray(n)`, n zero bytes, so a column of large integers (epoch milliseconds: about a terabyte per value) hung or ran out of memory. The binary test now answers for an integer without building it; the inferred types are unchanged.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
 
 ______________________________________________________________________
 

{execsql2-2.31.2 → execsql2-2.33.0}/docs/dev/adding_db_adapters.md

@@ -148,6 +148,7 @@ These are the instance attributes and methods you must configure correctly:
 | `schema_exists()`  | method            | **Must override.** Return `False` if schemas are not supported.                   |
 | `drop_table()`     | method            | **Must override.** Drop a table (used by IMPORT when creating fresh).             |
 | `populate_table()` | method            | **Must override.** Bulk-load rows from a generator (used by IMPORT).              |
+| `row_inserter()`   | method            | Override for a faster bulk path that stores exactly what `executemany()` stores.  |
 | `exec_cmd()`       | method            | Override if the DBMS can execute stored procedures or views as commands.          |
 
 Methods inherited from `Database` that you get for free include `execute()`, `cursor()`, `close()`, `commit()`, `rollback()`, `select_rowsource()`, `select_data()`, `schema_qualified_table_name()`, `quote_identifier()`, and `paramsubs()`.

{execsql2-2.31.2 → execsql2-2.33.0}/docs/dev/architecture.md

@@ -297,7 +297,7 @@ ______________________________________________________________________
 1. An `IMPORT` metacommand is dispatched to the appropriate handler.
 1. The handler instantiates the correct importer (e.g., `importers/csv.py`, `importers/ods.py`) based on the file type.
 1. The importer reads the file and yields rows.
-1. `Database.populate_table()` bulk-inserts the rows using `executemany()`, with optional progress bars.
+1. `Database.populate_table()` converts the rows and inserts them in batches of `import_row_buffer` rows, in one transaction, with optional progress bars. Each batch goes through the adapter's `row_inserter()`: `executemany()` by default, and multi-row `INSERT ... VALUES` on DuckDB, whose `executemany()` binds one row at a time.
 
 ### Export metadata
 

{execsql2-2.31.2 → execsql2-2.33.0}/docs/guides/formatter.md

@@ -272,7 +272,7 @@ select id,name,created_at from users where active = true order by name;
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.31.2
+    rev: v2.33.0
     hooks:
       - id: execsql-format
       - id: execsql-lint # optional: fail the commit on lint errors

{execsql2-2.31.2 → execsql2-2.33.0}/docs/reference/lint.md

@@ -158,7 +158,7 @@ The `execsql-lint` [pre-commit](https://pre-commit.com/) hook runs `execsql lint
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.31.2
+    rev: v2.33.0
     hooks:
       - id: execsql-lint
         args: [--ignore, V002] # optional: any execsql lint option

{execsql2-2.31.2 → execsql2-2.33.0}/docs/reference/metacommands.md

@@ -1766,7 +1766,7 @@ Evaluates whether or not the specified value is equal to zero. If the value is n
 METACOMMAND_ERROR()
 ```
 
-Evaluates whether the previous metacommand generated an error. This test for SQL errors will only be effective if the [METACOMMAND_ERROR_HALT OFF](#metacommanderrorhalt) metacommand has previously been issued. This conditional must be used in the first metacommand after any metacommand that might have encountered an error.
+Evaluates whether the previous metacommand generated an error. This test for SQL errors will only be effective if the [METACOMMAND_ERROR_HALT OFF](#metacommanderrorhalt) metacommand has previously been issued. This conditional must be used in the first metacommand after any metacommand that might have encountered an error: any metacommand that succeeds, even a WRITE, clears the flag. The error itself stays available until the next error: [$LAST_ERROR](substitution_vars.md#system_vars) holds the metacommand that failed and [$ERROR_MESSAGE](substitution_vars.md#system_vars) its message.
 
 
 ### *NEWER_DATE*
@@ -2109,7 +2109,7 @@ The [BREAK](#break) metacommand can be used to exit from a loop at any time.
 METACOMMAND_ERROR_HALT ON|OFF
 ```
 
-When METACOMMAND_ERROR_HALT is set to ON, which is the default, any errors that occur during execution of a metacommand will cause an error message to be displayed immediately, and *execsql* to halt. When METACOMMAND_ERROR_HALT is set to OFF, then metacommand errors will be ignored, but can be evaluated with the [IF METACOMMAND_ERROR](#metacommanderror) conditional.
+When METACOMMAND_ERROR_HALT is set to ON, which is the default, any errors that occur during execution of a metacommand will cause an error message to be displayed immediately, and *execsql* to halt. When METACOMMAND_ERROR_HALT is set to OFF, then metacommand errors will be ignored, but can be evaluated with the [IF METACOMMAND_ERROR](#metacommanderror) conditional; `$LAST_ERROR` and `$ERROR_MESSAGE` hold the failed metacommand and its message.
 
 The METACOMMAND_ERROR_HALT metacommand does not itself set or reset the internal flag that indicates whether a metacommand has encountered an error. This is so that a "METACOMMAND_ERROR_HALT Off" command can be used immediately after a potentially-failing metacommand and not alter the error flag that is set by the previous command. Specifically, constructions like this:
 
@@ -2356,6 +2356,8 @@ Set after every `PG_UPSERT` execution:
 | `$PG_UPSERT_ROWS_UPDATED` | integer | Total rows updated across all tables |
 | `$PG_UPSERT_ROWS_INSERTED` | integer | Total rows inserted across all tables |
 | `$PG_UPSERT_COMMITTED` | TRUE/FALSE | Were changes committed? |
+| `$PG_UPSERT_STATUS` | string | How the run ended: `committed`, `dry_run` (changes made, then rolled back because `COMMIT` was not given), `no_changes` (QA passed, nothing to insert or update), `qa_failed`, `cancelled` or `db_error`. Empty after `PG_UPSERT QA` and `PG_UPSERT CHECK`, which do not upsert. |
+| `$PG_UPSERT_ERROR` | string | The database error message when `$PG_UPSERT_STATUS` is `db_error`; otherwise empty |
 | `$PG_UPSERT_STAGING_SCHEMA` | string | Staging schema name used |
 | `$PG_UPSERT_BASE_SCHEMA` | string | Base schema name used |
 | `$PG_UPSERT_TABLES` | string | Comma-separated list of table names processed |
@@ -2380,7 +2382,7 @@ Set after every `PG_UPSERT` execution:
 
 ### QA failure behavior
 
-`PG_UPSERT`, `PG_UPSERT QA`, and `PG_UPSERT CHECK` do **not** raise an error when QA checks fail. The outcome is reported via `$PG_UPSERT_QA_PASSED` (and the per-table `$PG_UPSERT_TABLE_QA_PASSED`), so the script controls flow with `IF` or with execsql's standard error-halt configuration. When QA fails, the metacommand still runs `EXPORT_FAILURES`, populates all `$PG_UPSERT_*` substitution variables, and skips the upsert (no commit). Pair with `ASSERT` to halt explicitly:
+`PG_UPSERT`, `PG_UPSERT QA`, and `PG_UPSERT CHECK` do **not** raise an error when QA checks fail. The outcome is reported via `$PG_UPSERT_QA_PASSED` (and the per-table `$PG_UPSERT_TABLE_QA_PASSED`), so the script controls flow with `IF` or with execsql's standard error-halt configuration. When QA fails, the metacommand still runs `EXPORT_FAILURES`, populates all `$PG_UPSERT_*` substitution variables, and skips the upsert (no commit). To tell the ways a run can end apart, branch on `$PG_UPSERT_STATUS` rather than combining `$PG_UPSERT_QA_PASSED` with `$PG_UPSERT_COMMITTED`: a run with nothing to load (`no_changes`) and one the user cancelled (`cancelled`) both leave QA passed and nothing committed. Pair with `ASSERT` to halt explicitly:
 
 ```sql
 -- Soft failure: branch on the result, keep running.
@@ -2398,7 +2400,7 @@ Set after every `PG_UPSERT` execution:
 
 ### Database errors
 
-A database error during the run (a deadlock, a lock timeout, a constraint or trigger error the QA checks did not catch) is different from a QA failure. pg-upsert rolls the run back, and `PG_UPSERT` fails like any other metacommand: the driver's message is in `$ERROR_MESSAGE`, and the script halts unless `METACOMMAND_ERROR_HALT` is off. The `$PG_UPSERT_*` variables describe the failed run (`$PG_UPSERT_COMMITTED` is `FALSE` and the row counts are `0`), not an earlier `PG_UPSERT`.
+A database error during the run (a deadlock, a lock timeout, a constraint or trigger error the QA checks did not catch) is different from a QA failure. pg-upsert rolls the run back, and `PG_UPSERT` fails like any other metacommand: the driver's message is in `$ERROR_MESSAGE`, and the script halts unless `METACOMMAND_ERROR_HALT` is off. The `$PG_UPSERT_*` variables describe the failed run (`$PG_UPSERT_STATUS` is `db_error`, `$PG_UPSERT_ERROR` holds the message, `$PG_UPSERT_COMMITTED` is `FALSE` and the row counts are `0`), not an earlier `PG_UPSERT`. Cancelling at one of pg-upsert's own prompts is not an error: `PG_UPSERT` returns, with `$PG_UPSERT_STATUS` `cancelled`.
 
 The rollback ends the connection's transaction, so under `AUTOCOMMIT OFF` or inside `BEGIN BATCH`, statements run earlier on the same connection are rolled back with it.
 

{execsql2-2.31.2 → execsql2-2.33.0}/extras/vscode-execsql/package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "execsql-syntax",
-  "version": "2.31.2",
+  "version": "2.33.0",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "execsql-syntax",
-      "version": "2.31.2",
+      "version": "2.33.0",
       "license": "GPL-3.0-or-later",
       "dependencies": {
         "vscode-languageclient": "10.1.2"

{execsql2-2.31.2 → execsql2-2.33.0}/extras/vscode-execsql/package.json

@@ -2,7 +2,7 @@
   "name": "execsql-syntax",
   "displayName": "execsql",
   "description": "Syntax highlighting, lint findings, completion, hover, navigation, quick fixes and formatting for execsql scripts.",
-  "version": "2.31.2",
+  "version": "2.33.0",
   "publisher": "geocoug",
   "license": "GPL-3.0-or-later",
   "icon": "icon.png",

{execsql2-2.31.2 → execsql2-2.33.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "execsql2"
-version = "2.31.2"
+version = "2.33.0"
 description = "Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables."
 readme = { file = "README.md", content-type = "text/markdown" }
 license = { file = "LICENSE.txt" }
@@ -188,7 +188,7 @@ line-ending = "auto"
 exclude = ["*.md"]
 
 [tool.bumpversion]
-current_version = "2.31.2"
+current_version = "2.33.0"
 commit = true
 tag = true
 tag_name = "v{new_version}"

{execsql2-2.31.2 → execsql2-2.33.0}/src/execsql/db/base.py

@@ -665,6 +665,23 @@ class Database(ABC):
         self.execute(f"drop table if exists {tablename} cascade;")
         self.commit()
 
+    def row_inserter(self, curs: Any, table: str, columns: list[str]) -> Callable[[list[list[Any]]], None]:
+        """Return a function that inserts one batch of rows into *table* through *curs*.
+
+        *table* is schema-qualified and quoted; *columns* are the quoted
+        column names, in the order of each row's values.  IMPORT and COPY call
+        the function once per batch, all inside one transaction.  This one
+        runs ``executemany``.  An adapter may override it with a faster bulk
+        path only if that path stores exactly what ``executemany`` stores, for
+        every value type and column type: a faster load must not change data.
+        """
+        sql = f"insert into {table} ({','.join(columns)}) values ({self.paramsubs(len(columns))});"
+
+        def insert(rows: list[list[Any]]) -> None:
+            curs.executemany(sql, rows)
+
+        return insert
+
     def populate_table(
         self,
         schema_name: str | None,
@@ -740,6 +757,7 @@ class Database(ABC):
 
         def _import_loop(curs) -> int:
             nonlocal eof, total_rows, task_id
+            insert = self.row_inserter(curs, sq_name, columns)
             while True:
                 b = []
                 for _j in range(_state.conf.import_row_buffer):
@@ -809,7 +827,7 @@ class Database(ABC):
                                 b.append(row)
                 if len(b) > 0:
                     try:
-                        curs.executemany(sql, b)
+                        insert(b)
                     except ErrInfo:
                         raise
                     except Exception as e:

{execsql2-2.31.2 → execsql2-2.33.0}/src/execsql/db/duckdb.py

@@ -8,8 +8,12 @@ analytics databases via the ``duckdb`` package.  Corresponds to ``-t k``
 on the CLI.
 """
 
+import importlib.util
 import re
+import sys
 from collections import deque
+from collections.abc import Callable, Iterator
+from contextlib import contextmanager
 from pathlib import Path
 from typing import Any
 
@@ -18,7 +22,7 @@ from execsql.exceptions import ErrInfo
 from execsql.utils.errors import exception_desc, fatal_error, install_hint
 import execsql.state as _state
 
-__all__ = ["DuckDBDatabase"]
+__all__ = ["DuckDBDatabase", "insert_rows"]
 
 # Statements that return rows; any other statement DuckDB answers with a
 # one-column "Count" result (rows inserted, updated or deleted).
@@ -28,6 +32,53 @@ _QUERY_RX = re.compile(r"(?:SELECT|WITH|VALUES|FROM|TABLE|SHOW|DESCRIBE|SUMMARIZ
 _TRANSACTION_RX = re.compile(r"(?P<begin>BEGIN|START\s+TRANSACTION)\b|(?P<end>COMMIT|END|ROLLBACK|ABORT)\b", re.I)
 
 
+# Parameters bound by one INSERT ... VALUES statement in insert_rows(); a
+# wide table gets fewer rows per statement.
+_PARAMS_PER_STATEMENT = 10_000
+
+
+@contextmanager
+def _pandas_lookup_skipped() -> Iterator[None]:
+    """Make DuckDB's check for pandas fail at once while it binds rows.
+
+    The duckdb package asks Python's import system for ``pandas`` twice for
+    every row it binds, and when pandas is not installed nothing remembers
+    the answer, so each row searches ``sys.path`` again: about three
+    quarters of an INSERT's time.  A ``None`` in ``sys.modules`` makes that
+    import fail without the search.  It is set only when pandas is not
+    installed, so no import that could have succeeded fails, and it is
+    removed afterwards.
+    """
+    if "pandas" in sys.modules or importlib.util.find_spec("pandas") is not None:
+        yield
+        return
+    sys.modules["pandas"] = None  # type: ignore[assignment]
+    try:
+        yield
+    finally:
+        if "pandas" in sys.modules and sys.modules["pandas"] is None:
+            del sys.modules["pandas"]
+
+
+def insert_rows(curs: Any, table: str, columns: list[str], rows: list[Any]) -> None:
+    """Insert *rows* into *table* with multi-row ``INSERT ... VALUES`` statements.
+
+    *table* is quoted (and schema-qualified as needed); *columns* are the
+    quoted column names, in the order of each row's values.  DuckDB's
+    ``executemany`` runs one row at a time; binding many rows in one
+    statement is several times faster and stores the same values.
+    """
+    if not rows:
+        return
+    per_statement = max(1, _PARAMS_PER_STATEMENT // max(1, len(columns)))
+    head = f"insert into {table} ({','.join(columns)}) values "
+    row_marks = "(" + ",".join("?" * len(columns)) + ")"
+    with _pandas_lookup_skipped():
+        for start in range(0, len(rows), per_statement):
+            chunk = rows[start : start + per_statement]
+            curs.execute(head + ",".join([row_marks] * len(chunk)), [value for row in chunk for value in row])
+
+
 class _SharedCursor:
     """A cursor that runs on the database's one connection.
 
@@ -261,6 +312,14 @@ class DuckDBDatabase(Database):
             curs.execute(sql, params)
             return len(curs.fetchall()) > 0
 
+    def row_inserter(self, curs: Any, table: str, columns: list[str]) -> Callable[[list[list[Any]]], None]:
+        """Insert each batch with multi-row ``INSERT ... VALUES`` (:func:`insert_rows`)."""
+
+        def insert(rows: list[list[Any]]) -> None:
+            insert_rows(curs, table, columns, rows)
+
+        return insert
+
     def view_exists(self, view_name: str, schema_name: str | None = None) -> bool:
         """Return True if the named view exists in the DuckDB database."""
         # DuckDB information_schema has no 'views' table; views are listed in 'tables'

{execsql2-2.31.2 → execsql2-2.33.0}/src/execsql/exporters/duckdb.py

@@ -10,7 +10,6 @@ Requires the ``execsql2[duckdb]`` extra.
 
 import datetime as _datetime
 import decimal as _decimal
-import math
 from pathlib import Path
 from typing import Any
 
@@ -38,9 +37,9 @@ def export_duckdb(
         fatal_error(f"The duckdb module is required to export data in that format. {install_hint('duckdb')}")
         return
 
+    from execsql.db.duckdb import insert_rows
     from execsql.models import DataTable
 
-    chunksize = 10000
     # B07a/F026: tablename and catalog are substituted from user input.
     # Identifier-quote the table-name interpolation site and use
     # placeholders for the information_schema literal-value query.
@@ -70,15 +69,8 @@ def export_duckdb(
     curs.execute(sql)
     # Export all rows of data
     columns = [dbt_duckdb.quoted(col) for col in hdrs]
-    colspec = ",".join(columns)
-    paramspec = ",".join(("?",) * len(columns))
-    sql = f"insert into {qtable} ({colspec}) values ({paramspec});"
-    n_chunks = math.ceil(len(rowdata) / chunksize)
     curs.execute("BEGIN TRANSACTION;")
-    for i in range(n_chunks):
-        start = i * chunksize
-        end = start + chunksize
-        curs.executemany(sql, rowdata[start:end])
+    insert_rows(curs, qtable, columns, rowdata)
     curs.execute("COMMIT;")
     curs.close()
     ddb.close()

{execsql2-2.31.2 → execsql2-2.33.0}/src/execsql/metacommands/reference.py

@@ -1364,6 +1364,16 @@ _SYSTEM_VARIABLES: tuple[SystemVariable, ...] = (
         "TRUE or FALSE: whether the changes were committed.",
         "metacommands.md#pg_upsert",
     ),
+    SystemVariable(
+        "$PG_UPSERT_STATUS",
+        "How the run ended: committed, dry_run, no_changes, qa_failed, cancelled or db_error.",
+        "metacommands.md#pg_upsert",
+    ),
+    SystemVariable(
+        "$PG_UPSERT_ERROR",
+        "The database error message when $PG_UPSERT_STATUS is db_error; otherwise empty.",
+        "metacommands.md#pg_upsert",
+    ),
     SystemVariable("$PG_UPSERT_STAGING_SCHEMA", "Staging schema name used.", "metacommands.md#pg_upsert"),
     SystemVariable("$PG_UPSERT_BASE_SCHEMA", "Base schema name used.", "metacommands.md#pg_upsert"),
     SystemVariable("$PG_UPSERT_TABLES", "Comma-separated list of table names processed.", "metacommands.md#pg_upsert"),

{execsql2-2.31.2 → execsql2-2.33.0}/src/execsql/metacommands/upsert.py

@@ -185,6 +185,11 @@ def _set_subvars(result: Any) -> None:
     sv("$PG_UPSERT_ROWS_UPDATED", str(result.total_updated))
     sv("$PG_UPSERT_ROWS_INSERTED", str(result.total_inserted))
     sv("$PG_UPSERT_COMMITTED", str(result.committed).upper())
+    # How the run ended: committed, dry_run, no_changes, qa_failed, cancelled
+    # or db_error.  Empty for QA and CHECK, which do not run an upsert.
+    status = getattr(result, "status", None)
+    sv("$PG_UPSERT_STATUS", status.value if status is not None else "")
+    sv("$PG_UPSERT_ERROR", getattr(result, "error", None) or "")
     sv("$PG_UPSERT_STAGING_SCHEMA", result.staging_schema)
     sv("$PG_UPSERT_BASE_SCHEMA", result.base_schema)
     sv("$PG_UPSERT_TABLES", ", ".join(t.table_name for t in result.tables))
@@ -426,6 +431,14 @@ def _handle_pg_upsert_errors(fn: Any, metacommandline: str | None) -> Any:
     try:
         return fn()
     except UserCancelledError as exc:
+        # No result describes a run cancelled outside pg-upsert's own run();
+        # say what is known rather than leave an earlier run's values.
+        sv = _state.subvars.add_substitution
+        sv("$PG_UPSERT_STATUS", "cancelled")
+        sv("$PG_UPSERT_ERROR", "")
+        sv("$PG_UPSERT_COMMITTED", "FALSE")
+        sv("$PG_UPSERT_ROWS_UPDATED", "0")
+        sv("$PG_UPSERT_ROWS_INSERTED", "0")
         raise ErrInfo(
             "cmd",
             command_text=metacommandline,