execsql2 2.32.0__tar.gz → 2.34.0__tar.gz

{execsql2-2.32.0 → execsql2-2.34.0}/CHANGELOG.md

@@ -13,6 +13,40 @@ ______________________________________________________________________
 
 ______________________________________________________________________
 
+## [2.34.0] - 2026-10-18
+
+### Added
+
+- **`EXPORT … AS JSONL` and `EXPORT QUERY … AS JSONL`** write [JSON Lines](https://jsonlines.org/): one object per row, one row per line. `APPEND` adds rows to the file, which stays valid; `IMPORT … FROM JSON` reads it back.
+
+### Changed
+
+- **A value with a fraction loaded into an integer column** (`INTEGER`, `BIGINT`, `DECIMAL(10,0)`, …) of an existing table by `IMPORT` or `COPY` stops the load with an error that names the value and the column, and nothing is loaded. The database used to round it (`1.5` stored as `2`), or on SQL Server truncate it. A delimited file loaded into PostgreSQL or MySQL is not affected: the server reads it as written. Round the values in the source, or load them into a `DECIMAL` column with places for the fraction.
+- **MySQL / MariaDB sessions run in strict mode**: a SQL statement, `IMPORT` or `COPY` that gives a column a value MySQL can't store as given stops with MySQL's error, and nothing is loaded. MySQL used to store a changed value with a warning (`3000000000` as `2147483647`, `2024-02-30` as `0000-00-00`, `abcdef` in a `VARCHAR(3)` as `abc`). A timestamp ending in `Z` is one such value: write it with `+00:00`. To keep the old behavior in a script, run `SET SESSION sql_mode = 'ANSI';` after connecting.
+- **Scripts with many substitution variables run faster**: a statement or metacommand with no `!!`, `!'!` or `!"!` in it is no longer checked against every defined variable. With 500 variables each such line took about 0.1 ms longer; a 1,000-pass `LOOP` over `EXECUTE SCRIPT` with 200 variables now runs in half the time.
+
+### Fixed
+
+- **`IMPORT` and `COPY` into a table that already exists** convert each value by the type of the column it goes into, not the type guessed from the source data. A text column gets the value as written: `yes`, `1`, `01/05/2024`, `1e3` and `" 7 "` were stored as `true`, `true`, `2024-01-05 00:00:00`, `1000` and `7`. On MySQL a missing value in such a column is NULL, not `''`. `TO NEW` and `TO REPLACEMENT` tables are created and filled as before.
+- **`COPY` of a column of 0s and 1s into a PostgreSQL integer, numeric or float column** loads the numbers; it failed with "column … is of type integer but expression is of type boolean". The same goes for true/false values (`yes`/`no`, `t`/`f`) copied, or imported from a spreadsheet, JSON or Feather file, into such a column: they load as 1 and 0, as on the other databases.
+- **A missing date or timestamp loaded into MySQL by `IMPORT` or `COPY`** is stored as NULL; it was stored as `0000-00-00` (or `0` in a number column). Delimited files were not affected.
+
+______________________________________________________________________
+
+## [2.33.0] - 2026-10-16
+
+### Added
+
+- **`$PG_UPSERT_STATUS`** says how a `PG_UPSERT` ended (`committed`, `dry_run`, `no_changes`, `qa_failed`, `cancelled`, `db_error`), and **`$PG_UPSERT_ERROR`** holds the database error message. A run with nothing to load and a cancelled run used to look the same in the other variables.
+
+### Fixed
+
+- **A failed metacommand sets `$LAST_ERROR` and `$ERROR_MESSAGE`** with `METACOMMAND_ERROR_HALT OFF`, so a script can tell what failed, not only that `METACOMMAND_ERROR()` is true; an `ON ERROR_HALT` handler (including `ON ERROR_HALT WRITE "…!{$LAST_ERROR}!"`) now gets `$LAST_ERROR` too. The error is also written to `execsql.log`.
+- **`execsql.run()` reports a metacommand that failed under `METACOMMAND_ERROR_HALT OFF`** in `result.errors`, with `success` False, as it already did for a SQL error under `ERROR_HALT OFF`.
+- **`PG_UPSERT` cancelled from a prompt** no longer leaves an earlier run's `$PG_UPSERT_*` values (such as `$PG_UPSERT_COMMITTED` `TRUE`) in place.
+
+______________________________________________________________________
+
 ## [2.32.0] - 2026-10-16
 
 ### Changed

{execsql2-2.32.0 → execsql2-2.34.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: execsql2
-Version: 2.32.0
+Version: 2.34.0
 Summary: Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables.
 Project-URL: Homepage, https://execsql2.readthedocs.io/en/stable/
 Project-URL: Repository, https://github.com/geocoug/execsql
@@ -318,7 +318,7 @@ See the [full options reference](https://execsql2.readthedocs.io/en/stable/getti
 **Data movement**
 
 - Import data from CSV, TSV, JSON, Excel, OpenDocument, Feather, or Parquet files into a database table.
-- Export query results in 20+ formats including CSV, TSV, JSON, YAML, XML, HTML, Markdown, LaTeX, XLSX, OpenDocument, Feather, Parquet, HDF5, DuckDB, SQLite, plain text, and Jinja2 templates.
+- Export query results in 20+ formats including CSV, TSV, JSON, JSON Lines, YAML, XML, HTML, Markdown, LaTeX, XLSX, OpenDocument, Feather, Parquet, HDF5, DuckDB, SQLite, plain text, and Jinja2 templates.
 - Copy data between databases, including across different DBMS types.
 
 **Script logic**
@@ -441,7 +441,7 @@ execsql format --no-sql --in-place scripts/
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.32.0
+    rev: v2.34.0
     hooks:
       - id: execsql-format
       - id: execsql-lint

{execsql2-2.32.0 → execsql2-2.34.0}/README.md

@@ -198,7 +198,7 @@ See the [full options reference](https://execsql2.readthedocs.io/en/stable/getti
 **Data movement**
 
 - Import data from CSV, TSV, JSON, Excel, OpenDocument, Feather, or Parquet files into a database table.
-- Export query results in 20+ formats including CSV, TSV, JSON, YAML, XML, HTML, Markdown, LaTeX, XLSX, OpenDocument, Feather, Parquet, HDF5, DuckDB, SQLite, plain text, and Jinja2 templates.
+- Export query results in 20+ formats including CSV, TSV, JSON, JSON Lines, YAML, XML, HTML, Markdown, LaTeX, XLSX, OpenDocument, Feather, Parquet, HDF5, DuckDB, SQLite, plain text, and Jinja2 templates.
 - Copy data between databases, including across different DBMS types.
 
 **Script logic**
@@ -321,7 +321,7 @@ execsql format --no-sql --in-place scripts/
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.32.0
+    rev: v2.34.0
     hooks:
       - id: execsql-format
       - id: execsql-lint

{execsql2-2.32.0 → execsql2-2.34.0}/docs/about/divergence.md

@@ -43,20 +43,21 @@ ______________________________________________________________________
 | `PARQUET`         | Export query or table results to Apache Parquet via `polars`.                                                        |
 | `FEATHER`         | Export to Apache Feather/IPC via `polars` (upstream used `pandas` + `pyarrow`).                                      |
 | `YAML`            | Export query or table results as a YAML sequence of mappings via `PyYAML`.                                           |
+| `JSONL`           | Export query or table results as JSON Lines: one object per row, one row per line; `APPEND` adds rows (#7).          |
 | `MARKDOWN` / `MD` | Export query or table results as a GitHub-Flavored Markdown (GFM) pipe table. Pure Python, no optional dependencies. |
 | `XLSX`            | Export query or table results to an Excel XLSX workbook via `openpyxl` (single or multi-sheet).                      |
 
 ### Metacommands
 
-| Metacommand             | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
-| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `ASSERT`                | Evaluate a condition and raise an error (halting the script) if it is false. Supports all IF conditions. Optional quoted failure message. Skipped in false IF blocks.                                                                                                                                                                                                                                                                                          |
-| `BREAKPOINT`            | Pause script execution and drop into an interactive debug REPL. See [Debugging](#debugging) below for full details.                                                                                                                                                                                                                                                                                                                                            |
-| `CONFIG SHOW_PROGRESS`  | Enable the Rich progress bar for IMPORT operations at runtime.                                                                                                                                                                                                                                                                                                                                                                                                 |
-| `CONFIG LOG_SQL`        | Enable SQL query audit logging — writes executed SQL to the log file.                                                                                                                                                                                                                                                                                                                                                                                          |
-| `PG_UPSERT`             | QA-checked, FK-dependency-ordered upserts from staging to base schema on PostgreSQL via the optional [pg-upsert](https://pg-upsert.readthedocs.io/) dependency. Modes: `PG_UPSERT` (full pipeline), `PG_UPSERT QA` (checks only), `PG_UPSERT CHECK` (schema only). QA checks only the rows the given `METHOD` would write. QA failure does not raise — the result is reported via `$PG_UPSERT_QA_PASSED` so the caller's `IF` / `ASSERT` decides control flow. |
-| `IMPORT … FROM JSON`    | Import a JSON file (array of objects or [JSON Lines](https://jsonlines.org/)) into a database table. Nested objects are flattened with dot-separated column names; arrays are stored as JSON strings.                                                                                                                                                                                                                                                          |
-| `SHOW SCRIPTS [<name>]` | Without a name, lists all registered SCRIPT definitions with parameter signatures and source locations. With a name, shows detail including parameters (with defaults), source file/line range, and docstring.                                                                                                                                                                                                                                                 |
+| Metacommand             | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
+| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `ASSERT`                | Evaluate a condition and raise an error (halting the script) if it is false. Supports all IF conditions. Optional quoted failure message. Skipped in false IF blocks.                                                                                                                                                                                                                                                                                                                                      |
+| `BREAKPOINT`            | Pause script execution and drop into an interactive debug REPL. See [Debugging](#debugging) below for full details.                                                                                                                                                                                                                                                                                                                                                                                        |
+| `CONFIG SHOW_PROGRESS`  | Enable the Rich progress bar for IMPORT operations at runtime.                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| `CONFIG LOG_SQL`        | Enable SQL query audit logging — writes executed SQL to the log file.                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+| `PG_UPSERT`             | QA-checked, FK-dependency-ordered upserts from staging to base schema on PostgreSQL via the optional [pg-upsert](https://pg-upsert.readthedocs.io/) dependency. Modes: `PG_UPSERT` (full pipeline), `PG_UPSERT QA` (checks only), `PG_UPSERT CHECK` (schema only). QA checks only the rows the given `METHOD` would write. QA failure does not raise — the result is reported via `$PG_UPSERT_QA_PASSED` so the caller's `IF` / `ASSERT` decides control flow; `$PG_UPSERT_STATUS` says how the run ended. |
+| `IMPORT … FROM JSON`    | Import a JSON file (array of objects or [JSON Lines](https://jsonlines.org/)) into a database table. Nested objects are flattened with dot-separated column names; arrays are stored as JSON strings.                                                                                                                                                                                                                                                                                                      |
+| `SHOW SCRIPTS [<name>]` | Without a name, lists all registered SCRIPT definitions with parameter signatures and source locations. With a name, shows detail including parameters (with defaults), source file/line range, and docstring.                                                                                                                                                                                                                                                                                             |
 
 ### SCRIPT Enhancements
 
@@ -228,6 +229,12 @@ Upstream opened MySQL and MariaDB connections as `latin1`, and execsql2 did the
 
 The default is now `utf8mb4`. This does not change stored data or column charsets: the connection charset governs only how text moves between client and server, and MySQL transcodes to and from each column's own charset. A latin1 column read over a utf8mb4 connection returns the same string it always did. `-e latin1` restores the upstream behavior.
 
+### MySQL Strict Mode
+
+Upstream set `sql_mode='ANSI'` on every MySQL and MariaDB connection. That replaces the server's whole mode, and `ANSI` leaves out `STRICT_TRANS_TABLES`, so MySQL stored a changed value with a warning instead of raising an error: `3000000000` in an `INTEGER` column became `2147483647`, `2024-02-30` became `0000-00-00`, and `abcdef` in a `VARCHAR(3)` became `abc`. That applied to plain SQL statements, `IMPORT` and `COPY` alike (#101).
+
+The mode is now `ANSI,STRICT_TRANS_TABLES`, so such a value is an error, as in the `mysql` client with the server's default settings. The mode stays fixed rather than added to the server's own, because execsql's quoting of literals relies on backslash escapes being on (no `NO_BACKSLASH_ESCAPES`). `LOAD DATA LOCAL`, which `IMPORT` uses for a delimited file, ignores strict mode, so `IMPORT` checks its warnings and rolls the load back if MySQL changed a value. `SET SESSION sql_mode = 'ANSI';` in a script restores the upstream behavior for that session, `LOAD DATA` included.
+
 ### Database Support Tiers
 
 Upstream documented all nine DBMS adapters with equal confidence. execsql2 splits them by what is actually verified: **Supported** (PostgreSQL, MySQL/MariaDB, MS SQL Server, SQLite, DuckDB) run against a live server or a real database file on every CI run, while **best effort** (MS Access, Firebird, Oracle, ODBC DSN) are carried forward from the monolith and are exercised nowhere in CI.
@@ -349,7 +356,7 @@ These are behavioral changes driven by security or correctness issues in the ups
 | `WRITE` after `CD`                                                 | A relative `WRITE ... TO` path is resolved against the working directory at the time of the `WRITE`, so it follows an earlier `CD` the way `EXPORT` and `IMPORT` already did. Upstream wrote it relative to the directory execsql started in, because its writer subprocess never saw the `CD`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
 | Substitution variable cycles                                       | 100-iteration limit prevents infinite loops on cyclic variable references.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
 | Script location in error messages                                  | `ErrInfo.script_file` and `script_line_no` are now populated via `stamp_errinfo()` so error output includes "Line N of script foo.sql" context — restoring behavior present in the monolith.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
-| `$ERROR_MESSAGE` not updated                                       | `$ERROR_MESSAGE` is now set on every error path: `exit_now()`, non-halting SQL errors, and non-halting metacommand errors. Previously it was initialized to `""` and never changed.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
+| `$ERROR_MESSAGE` not updated                                       | `$ERROR_MESSAGE` is now set on every error path: `exit_now()`, non-halting SQL errors, and non-halting metacommand errors. Previously it was initialized to `""` and never changed. `$LAST_ERROR` also holds a failed metacommand, with `METACOMMAND_ERROR_HALT OFF` and in an `ON ERROR_HALT` handler, as the documentation describes; upstream set it only for SQL errors and unknown metacommands.                                                                                                                                                                                                                                                                                                                                                                                                              |
 | Metacommand error message lost                                     | When `halt_on_metacommand_err` is `ON`, the original handler `ErrInfo` is now re-raised; the generic "Unknown metacommand" message no longer replaces the specific error from the handler.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
 | Empty script name in error msg                                     | `_execute_script_direct()` and `_execute_script_textual_console()` no longer append "in script , line 0" to uncaught-exception messages when `current_script_line()` returns an empty string.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
 | `PROMPT COMPARE` diff comparison                                   | Diff engine uses native Python equality instead of string comparison — numeric types, Decimals, and booleans compare correctly. `None` is distinguished from empty string. Columns are matched by name (not position), key columns are excluded from comparison, and duplicate PKs keep the first row.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
@@ -436,6 +443,9 @@ These are behavioral changes driven by security or correctness issues in the ups
 | XLSX export wrote `numeric` columns as text                        | `_cell_value()` passed `int` and `float` straight to openpyxl but let every other type fall through to `str()`. PostgreSQL, MySQL, and DuckDB all return `Decimal` for `numeric`/`decimal` columns, so measurement values arrived in Excel as text cells — invisible in a screenshot, useless in a formula, a sort, or a chart. openpyxl lists `Decimal` in its own `NUMERIC_TYPES` and stores it as a real numeric cell, so the conversion was never needed. Found by exporting from a live PostgreSQL container and checking `cell.data_type` rather than `cell.value`.                                                                                                                                                                                                                                          |
 | `EMAIL` SMTP login without a password                              | Upstream calls `smtplib.login()` with only a username when `[email] password` and `enc_password` are both unset, which raises `TypeError` from the stdlib — `login()` has no single-argument form. execsql2 reports a configuration error naming the missing password instead.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
 | Type scan of large integers                                        | Working out a column's type tested each integer against the binary type by building `bytearray(n)`, n zero bytes, so a column of large integers (epoch milliseconds: about a terabyte per value) hung or ran out of memory. The binary test now answers for an integer without building it; the inferred types are unchanged.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+| Values loaded into an existing table                               | `IMPORT` and `COPY` converted every value to the type guessed from the source data, even when the table was already there with its own column types, so a text column got a different string than the source held (`yes` and `1` stored as `true`, `01/05/2024` as `2024-01-05 00:00:00`, `1e3` as `1000`), and on PostgreSQL a column of 0s and 1s copied into an integer column failed (the guess was boolean). For a table that exists, the target column's type now decides: a text column gets the value as written, a number column gets true/false values as 1 and 0, and other columns are converted as before. Each adapter reads the column types from its catalog (`pg_attribute`, `SHOW COLUMNS`, `sys.columns`) or its driver's description of the table. `TO NEW` / `REPLACEMENT` are unchanged.     |
+| Missing dates loaded into MySQL                                    | `IMPORT` and `COPY` sent a missing value in a column scanned as a date or timestamp as `''`, so MySQL stored `0000-00-00` (or `0` in a number column) instead of NULL. It is sent as NULL.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+| Fractions loaded into an integer column                            | A value with a fraction going into an integer column (including `DECIMAL(p,0)`, `NUMERIC(p)` and Oracle `NUMBER(p)`) of an existing table is an error naming the value and the column. Upstream sent the number and the database rounded it (`1.5` → `2`; SQL Server truncated it to `1`).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
 
 ______________________________________________________________________
 

{execsql2-2.32.0 → execsql2-2.34.0}/docs/api/db.md

@@ -8,6 +8,12 @@ If you are adding support for a new database, start with the [Adding Database Ad
 
 ::: execsql.db.base
 
+## Target column types
+
+`IMPORT` and `COPY` into a table that already exists convert each value by the type of the column it goes into (`Database.column_kinds()`), not by the type scanned from the source data.
+
+::: execsql.db.target_types
+
 ## Database factory
 
 Convenience constructors used internally to create typed `Database` instances. The CLI calls the appropriate factory function based on the `-t` flag value. Each factory validates its arguments (e.g., checking that a file exists) before constructing and returning the adapter.

{execsql2-2.32.0 → execsql2-2.34.0}/docs/dev/adding_db_adapters.md

@@ -133,23 +133,24 @@ class MyDBDatabase(Database):
 
 These are the instance attributes and methods you must configure correctly:
 
-| Attribute / Method | Type              | Purpose                                                                           |
-| ------------------ | ----------------- | --------------------------------------------------------------------------------- |
-| `self.type`        | `DbType`          | DBMS type token (e.g., `dbt_sqlite`). Controls quoting and type-mapping.          |
-| `self.paramstr`    | `str`             | SQL parameter placeholder: `"?"` (most drivers) or `"%s"` (psycopg, PyMySQL).     |
-| `self.encoding`    | `str`             | Database character encoding. Detect from the database on connect if possible.     |
-| `self.autocommit`  | `bool`            | `True` means the driver commits automatically; `False` requires explicit commits. |
-| `self.conn`        | driver connection | Set in `open_db()`.                                                               |
-| `open_db()`        | method            | **Must override.** Establish the connection and assign `self.conn`.               |
-| `table_exists()`   | method            | **Must override.** Query the DBMS catalog. Use parameterized queries.             |
-| `column_exists()`  | method            | **Must override.** Check column presence.                                         |
-| `table_columns()`  | method            | **Must override.** Return column names for a table.                               |
-| `view_exists()`    | method            | **Must override.** Check view presence.                                           |
-| `schema_exists()`  | method            | **Must override.** Return `False` if schemas are not supported.                   |
-| `drop_table()`     | method            | **Must override.** Drop a table (used by IMPORT when creating fresh).             |
-| `populate_table()` | method            | **Must override.** Bulk-load rows from a generator (used by IMPORT).              |
-| `row_inserter()`   | method            | Override for a faster bulk path that stores exactly what `executemany()` stores.  |
-| `exec_cmd()`       | method            | Override if the DBMS can execute stored procedures or views as commands.          |
+| Attribute / Method | Type              | Purpose                                                                                                                                                                             |
+| ------------------ | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `self.type`        | `DbType`          | DBMS type token (e.g., `dbt_sqlite`). Controls quoting and type-mapping.                                                                                                            |
+| `self.paramstr`    | `str`             | SQL parameter placeholder: `"?"` (most drivers) or `"%s"` (psycopg, PyMySQL).                                                                                                       |
+| `self.encoding`    | `str`             | Database character encoding. Detect from the database on connect if possible.                                                                                                       |
+| `self.autocommit`  | `bool`            | `True` means the driver commits automatically; `False` requires explicit commits.                                                                                                   |
+| `self.conn`        | driver connection | Set in `open_db()`.                                                                                                                                                                 |
+| `open_db()`        | method            | **Must override.** Establish the connection and assign `self.conn`.                                                                                                                 |
+| `table_exists()`   | method            | **Must override.** Query the DBMS catalog. Use parameterized queries.                                                                                                               |
+| `column_exists()`  | method            | **Must override.** Check column presence.                                                                                                                                           |
+| `table_columns()`  | method            | **Must override.** Return column names for a table.                                                                                                                                 |
+| `view_exists()`    | method            | **Must override.** Check view presence.                                                                                                                                             |
+| `schema_exists()`  | method            | **Must override.** Return `False` if schemas are not supported.                                                                                                                     |
+| `drop_table()`     | method            | **Must override.** Drop a table (used by IMPORT when creating fresh).                                                                                                               |
+| `populate_table()` | method            | **Must override.** Bulk-load rows from a generator (used by IMPORT).                                                                                                                |
+| `row_inserter()`   | method            | Override for a faster bulk path that stores exactly what `executemany()` stores.                                                                                                    |
+| `column_kinds()`   | method            | Override if the driver's description of a table does not tell its column types apart (see `db/target_types.py`); IMPORT and COPY into an existing table convert each value by them. |
+| `exec_cmd()`       | method            | Override if the DBMS can execute stored procedures or views as commands.                                                                                                            |
 
 Methods inherited from `Database` that you get for free include `execute()`, `cursor()`, `close()`, `commit()`, `rollback()`, `select_rowsource()`, `select_data()`, `schema_qualified_table_name()`, `quote_identifier()`, and `paramsubs()`.
 

{execsql2-2.32.0 → execsql2-2.34.0}/docs/dev/architecture.md

@@ -264,6 +264,8 @@ ______________________________________________________________________
 
 The base class provides shared implementations for `execute()`, `select_data()`, `select_rowsource()`, `select_rowdict()`, `commit()`, `rollback()`, and `populate_table()`.
 
+`populate_table()` converts each value to the type scanned from the source (`models.DataTable`). For a table that was already there (`existing_table=True`, passed by IMPORT and COPY), `column_kinds()` reports the target columns' types and `db/target_types.py` picks each column's conversion instead: a text column gets the value as given, an integer column whole numbers or an error. `column_kinds()` reads the driver's description of `select * from <table> where 1=0` by default; PostgreSQL, MySQL and SQL Server read their catalogs, where the description cannot tell some types apart. `tests/db/test_existing_table_loads.py` compares what every value type stores in every column type with the record in `tests/data/existing_table_loads/`.
+
 ### DatabasePool
 
 `DatabasePool` is a dict-like container mapping string aliases to open `Database` instances. It is the canonical `_state.dbs` object. Key operations:

{execsql2-2.32.0 → execsql2-2.34.0}/docs/guides/formatter.md

@@ -272,7 +272,7 @@ select id,name,created_at from users where active = true order by name;
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.32.0
+    rev: v2.34.0
     hooks:
       - id: execsql-format
       - id: execsql-lint # optional: fail the commit on lint errors

{execsql2-2.32.0 → execsql2-2.34.0}/docs/guides/sql_syntax.md

@@ -4,6 +4,26 @@
 
 When execsql connects to a SQL Server or MySQL/MariaDB database, it automatically configures the DBMS to expect ANSI-compatible SQL, to allow the use of more standards-compliant, and thus consistent, SQL. In particular, for MySQL/MariaDB, note that the double-quote character, rather than the backtick character, must be used to quote table, schema, and column names, and only the apostrophe can be used to quote character data.
 
+### MySQL strict mode { #mysql_strict_mode }
+
+On MySQL/MariaDB, execsql sets the session's `sql_mode` to `ANSI,STRICT_TRANS_TABLES`. A value that MySQL can't store as given is an error, for a SQL statement, [IMPORT](../reference/metacommands.md#import) or [COPY](../reference/metacommands.md#copy):
+
+```sql
+create table m (n integer, d date, s varchar(3));
+insert into m values (3000000000, '2024-02-30', 'abcdef');
+-- ERROR 1264 (22003): Out of range value for column 'n' at row 1
+```
+
+Without strict mode, MySQL would have stored `2147483647`, `0000-00-00` and `abc` with only a warning. This is how the `mysql` client behaves with the server's default settings. An IMPORT of a delimited file that MySQL reads with `LOAD DATA` is held to the same rule: if MySQL warns that it changed a value, the IMPORT stops and nothing is loaded.
+
+MySQL in strict mode does not read a trailing `Z` as a time zone: write `2024-01-05T10:00:00Z` as `2024-01-05 10:00:00+00:00`. A decimal with more places than its column has (`1.2345` into `DECIMAL(10,3)`) is still rounded, as MySQL does in strict mode.
+
+A script that relies on MySQL's changed values can turn strict mode off for the rest of its session:
+
+```sql
+SET SESSION sql_mode = 'ANSI';
+```
+
 ## Implicit Commits
 
 By default, execsql immediately commits all SQL statements. The [AUTOCOMMIT](../reference/metacommands.md#autocommit) metacommand can be used to turn off automatic commits, and the [BATCH](../reference/metacommands.md#batch) metacommand can be used to delay commits until the end of a batch. [IMPORT](../reference/metacommands.md#import) and [COPY](../reference/metacommands.md#copy) are the only metacommands that change data, and they also automatically commit their changes when complete (unless [AUTOCOMMIT](../reference/metacommands.md#autocommit) has been turned off). If a new table is created with either of these metacommands (through the use of the NEW or REPLACEMENT keywords), the CREATE TABLE statement will not be committed separately from the data addition, except when using Firebird. Thus, if an error occurs during addition of the data, the new target table will not exist---except when using Firebird.

{execsql2-2.32.0 → execsql2-2.34.0}/docs/index.md

@@ -9,7 +9,7 @@
 You can use *execsql* to:
 
 - Import data from text files and spreadsheets into a database.
-- Export tables and views to ~30 data formats including CSV, OpenDocument spreadsheets, XLSX, HTML, JSON, YAML, Markdown, LaTeX, XML, Parquet, Feather, HDF5, SQLite, and DuckDB.
+- Export tables and views to ~30 data formats including CSV, OpenDocument spreadsheets, XLSX, HTML, JSON, JSON Lines, YAML, Markdown, LaTeX, XML, Parquet, Feather, HDF5, SQLite, and DuckDB.
 - Copy data between different databases, even databases using different DBMSs.
 - Display tables or views on the terminal or in a GUI dialog window.
 - Export data using template processors to produce non-tabular output with customized format and contents.

{execsql2-2.32.0 → execsql2-2.34.0}/docs/reference/lint.md

@@ -158,7 +158,7 @@ The `execsql-lint` [pre-commit](https://pre-commit.com/) hook runs `execsql lint
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.32.0
+    rev: v2.34.0
     hooks:
       - id: execsql-lint
         args: [--ignore, V002] # optional: any execsql lint option

{execsql2-2.32.0 → execsql2-2.34.0}/docs/reference/metacommands.md

@@ -889,6 +889,8 @@ Copies the data from a data table or view in one database to a data table in a s
 
 The second (destination) table must have column names that are identical to the names of the columns in the first (source) table. The second table may have additional columns; if it does, they will not be affected and their names don't matter. The data types in the columns to be copied must be compatible, though not necessarily identical. The order of the columns in the two tables does not have to be identical.
 
+When neither "NEW" nor "REPLACEMENT" is used, each value is converted for the destination column it goes into, as for [IMPORT into an existing table](#import_existing_table_types): a text column gets the value as the source database returned it (an integer `1` is stored as `1`, a date as `2024-01-05`), an integer column gets whole numbers and stops the COPY with an error on a value with a fraction instead of rounding it, and a number column gets a column of 0s and 1s, or of true/false values, as 1 and 0.
+
 If the "NEW" keyword is used, the destination table will be automatically created with column names and data types that are compatible with the first (source) table. The data types used for the columns in the newly created table will be determined by a scan of all of the data in the first table, but may not exactly match those in the first table. A text column stays text, whatever its values look like: `00123`, `1`/`0` and `2024-01-05` are copied as written, not as numbers, booleans or dates. The source is read once; while its rows are scanned they are held in a temporary file, so a large copy does not need memory for the whole table. If the destination table already exists when the "NEW" keyword is used, the copy stops before creating anything, with the error `Table <name> already exists`; a name without a schema is looked for only in the schema the table would be created in (on MySQL, the current database)
 
 If the "REPLACEMENT" keyword is used, the destination table will also be created to be compatible with the source table, but any existing destination table of the same name will be dropped first. This replacement is not atomic: if the drop succeeds but a later create or load step fails, the old destination table has already been removed. If the drop fails, *execsql* stops the COPY command before creating or loading the replacement table. *execsql* uses a "drop table" statement to drop an existing destination table, and this statement may not succeed if there are dependencies on that table (see the discussion of [implicit drop table
@@ -1165,6 +1167,15 @@ JSON
 :   [Javascript Object Notation](http://www.json.org/). The data table is represented as an array of JSON objects, where each object represents a row of the table. Each row is represented as a set of key:value pairs, with column names used as the keys. No description text will be included in the output even if it is provided.
 
 
+JSONL
+
+:   [JSON Lines](https://jsonlines.org/): one JSON object per row, one row per line, with no surrounding array. Values are written as for JSON (dates and times as text, NULL as `null`); a line break inside a value is written as `\n`, so each row stays on one line. If the "APPEND" keyword is used, the rows are added to the end of the file, which stays valid JSON Lines (appending JSON instead leaves two arrays in one file). A query that returns no rows writes an empty file. The file can be read back with [IMPORT … FROM JSON](#import). No description text will be included in the output even if it is provided:
+
+    ```
+    -- !x! EXPORT QUERY <<select * from events where day = current_date;>> APPEND TO events.jsonl AS JSONL
+    ```
+
+
 JSON_TS or JSON_TABLESCHEMA
 
 :   [JSON Table Schema](https://frictionlessdata.io/specs/table-schema/). The data themselves are not exported. Ordinarily this form would be used in conjunction with an additional export of the data to a CSV file. The column type descriptions in the output are derived by scanning the entire table to evaluate the data type of each column. If a "NOTYPE" keyword is included before the "DESCRIPTION" keyword, this scanning is not done, and no type information is included in the output file.:
@@ -1766,7 +1777,7 @@ Evaluates whether or not the specified value is equal to zero. If the value is n
 METACOMMAND_ERROR()
 ```
 
-Evaluates whether the previous metacommand generated an error. This test for SQL errors will only be effective if the [METACOMMAND_ERROR_HALT OFF](#metacommanderrorhalt) metacommand has previously been issued. This conditional must be used in the first metacommand after any metacommand that might have encountered an error.
+Evaluates whether the previous metacommand generated an error. This test for SQL errors will only be effective if the [METACOMMAND_ERROR_HALT OFF](#metacommanderrorhalt) metacommand has previously been issued. This conditional must be used in the first metacommand after any metacommand that might have encountered an error: any metacommand that succeeds, even a WRITE, clears the flag. The error itself stays available until the next error: [$LAST_ERROR](substitution_vars.md#system_vars) holds the metacommand that failed and [$ERROR_MESSAGE](substitution_vars.md#system_vars) its message.
 
 
 ### *NEWER_DATE*
@@ -1945,7 +1956,7 @@ The syntax for importing data from a [JSON](https://www.json.org/) file is:
 IMPORT TO [NEW|REPLACEMENT] <table_name> FROM JSON <file_name>
 ```
 
-The JSON file must contain either a JSON array of objects (`[{…}, …]`) or [JSON Lines](https://jsonlines.org/) (JSONL, one object per line). Nested objects are flattened with dot-separated column names (e.g., an object `{"address": {"city": "Portland"}}` produces column `address.city`). Nested arrays within objects are stored as JSON strings. Records with different keys produce a superset of columns — missing keys become NULL. The parser auto-detects which form is present by peeking at the first non-whitespace character (`[` → array, `{` → JSON Lines); the file extension is not consulted.
+The JSON file must contain either a JSON array of objects (`[{…}, …]`) or [JSON Lines](https://jsonlines.org/) (JSONL, one object per line). Nested objects are flattened with dot-separated column names (e.g., an object `{"address": {"city": "Portland"}}` produces column `address.city`). Nested arrays within objects are stored as JSON strings. Records with different keys produce a superset of columns — missing keys become NULL. The parser auto-detects which form is present by peeking at the first non-whitespace character (`[` → array, `{` → JSON Lines); the file extension is not consulted. A file written by [EXPORT … AS JSONL](#export) is read as JSON Lines.
 
 Column names in the input must be valid for the DBMS in use.
 
@@ -1968,7 +1979,15 @@ A data line with fewer values than the header has NULL for the values left off t
 
 The rows are loaded in one transaction: if the load fails part-way (a constraint violation, a value of the wrong type, a row with too many columns), none of the file's rows are kept, on every DBMS. The exception is an ODBC DSN whose driver can only autocommit, which keeps the rows loaded before the failure; *execsql* warns once on such a connection.
 
-If neither the NEW or REPLACEMENT keywords are used, the table must exist, must have column names identical to those in the input data, and columns must have data types that are compatible with (though not necessarily identical to) those in the input data. If neither the NEW or REPLACEMENT keywords are used, the input data is not scanned to determine the data type of each column.
+<a id="import_existing_table_types"></a>
+If neither the NEW or REPLACEMENT keywords are used, the table must exist, must have column names identical to those in the input data, and columns must have data types that are compatible with (though not necessarily identical to) those in the input data. Each value is converted for the column it goes into:
+
+- A text column (`CHAR`, `VARCHAR`, `TEXT`, …) gets the value as written in the file: `yes`, `00701`, `01/05/2024` and `1e3` are stored as those strings. A missing value is NULL.
+- An integer column (`INTEGER`, `BIGINT`, `DECIMAL(10,0)`, `NUMERIC(10)`, Oracle `NUMBER(10)`, …) gets whole numbers: `7`, `+7`, `7.0` and `7e0` load as 7. A value with a fraction, such as `1.5`, stops the IMPORT with an error that names the value and the column, and nothing is loaded; it is not rounded.
+- A number column gets a column of true/false values (`yes`/`no`, `t`/`f`, `1`/`0`; see [BOOLEAN_INT](#boolean_int) and [BOOLEAN_WORDS](#boolean_words)) as 1 and 0.
+- Any other column (a date, a timestamp, a Boolean, a decimal with places for a fraction) gets the value converted to the data type found by scanning that column of the input, as with NEW, and the database converts it to the column's type.
+
+On PostgreSQL and MySQL a delimited text file goes to the server's own loader (`COPY`, `LOAD DATA`, or on PostgreSQL an `INSERT` of each value as text), and the server converts each value as written; the rules above apply to the other file formats, and to delimited files on the other DBMSs. On MySQL a value that the server can't store as given (out of range, an invalid date, text longer than the column) stops the IMPORT with an error, and nothing is loaded; see [MySQL strict mode](../guides/sql_syntax.md#mysql_strict_mode).
 
 If a table is scanned to determine data types, any column that is completely empty (all null) will be created with the text data type. This provides the greatest flexibility for subsequent addition of data to the table. However, if that column ought to have a different data type, and a WHERE clause is applied to that column assuming a different data type, the DBMS may report an error because of incomparable data types.
 
@@ -2032,7 +2051,7 @@ When MS-Excel saves an OpenDocument spreadsheet, it may create an additional emp
 Because the data addition to the target table is always committed except when [AUTOCOMMIT](#autocommit) is off, the IMPORT metacommand should be used with care within transactions that are managed with explicit SQL statements or with [BATCHes](#batch).
 
 
-In general, if an error occurs while importing data, none of the new data should be in the target table (the operation is not committed). However, MySQL/MariaDB may issue messages about data type incompatibility to the standard error device (ordinarily the terminal), yet load some or all of the data. If the NEW or REPLACEMENT keywords are used, depending on the DBMS and where the error occurred, the target table may be created even if the data are not loaded.
+In general, if an error occurs while importing data, none of the new data should be in the target table (the operation is not committed). On MySQL/MariaDB a value that the server would change to store it is an error too, unless the script has turned off [strict mode](../guides/sql_syntax.md#mysql_strict_mode). If the NEW or REPLACEMENT keywords are used, depending on the DBMS and where the error occurred, the target table may be created even if the data are not loaded.
 
 The name, size, and date of the IMPORTed file are written to the [execsql.log](../guides/logging.md#logging) file.
 
@@ -2109,7 +2128,7 @@ The [BREAK](#break) metacommand can be used to exit from a loop at any time.
 METACOMMAND_ERROR_HALT ON|OFF
 ```
 
-When METACOMMAND_ERROR_HALT is set to ON, which is the default, any errors that occur during execution of a metacommand will cause an error message to be displayed immediately, and *execsql* to halt. When METACOMMAND_ERROR_HALT is set to OFF, then metacommand errors will be ignored, but can be evaluated with the [IF METACOMMAND_ERROR](#metacommanderror) conditional.
+When METACOMMAND_ERROR_HALT is set to ON, which is the default, any errors that occur during execution of a metacommand will cause an error message to be displayed immediately, and *execsql* to halt. When METACOMMAND_ERROR_HALT is set to OFF, then metacommand errors will be ignored, but can be evaluated with the [IF METACOMMAND_ERROR](#metacommanderror) conditional; `$LAST_ERROR` and `$ERROR_MESSAGE` hold the failed metacommand and its message.
 
 The METACOMMAND_ERROR_HALT metacommand does not itself set or reset the internal flag that indicates whether a metacommand has encountered an error. This is so that a "METACOMMAND_ERROR_HALT Off" command can be used immediately after a potentially-failing metacommand and not alter the error flag that is set by the previous command. Specifically, constructions like this:
 
@@ -2356,6 +2375,8 @@ Set after every `PG_UPSERT` execution:
 | `$PG_UPSERT_ROWS_UPDATED` | integer | Total rows updated across all tables |
 | `$PG_UPSERT_ROWS_INSERTED` | integer | Total rows inserted across all tables |
 | `$PG_UPSERT_COMMITTED` | TRUE/FALSE | Were changes committed? |
+| `$PG_UPSERT_STATUS` | string | How the run ended: `committed`, `dry_run` (changes made, then rolled back because `COMMIT` was not given), `no_changes` (QA passed, nothing to insert or update), `qa_failed`, `cancelled` or `db_error`. Empty after `PG_UPSERT QA` and `PG_UPSERT CHECK`, which do not upsert. |
+| `$PG_UPSERT_ERROR` | string | The database error message when `$PG_UPSERT_STATUS` is `db_error`; otherwise empty |
 | `$PG_UPSERT_STAGING_SCHEMA` | string | Staging schema name used |
 | `$PG_UPSERT_BASE_SCHEMA` | string | Base schema name used |
 | `$PG_UPSERT_TABLES` | string | Comma-separated list of table names processed |
@@ -2380,7 +2401,7 @@ Set after every `PG_UPSERT` execution:
 
 ### QA failure behavior
 
-`PG_UPSERT`, `PG_UPSERT QA`, and `PG_UPSERT CHECK` do **not** raise an error when QA checks fail. The outcome is reported via `$PG_UPSERT_QA_PASSED` (and the per-table `$PG_UPSERT_TABLE_QA_PASSED`), so the script controls flow with `IF` or with execsql's standard error-halt configuration. When QA fails, the metacommand still runs `EXPORT_FAILURES`, populates all `$PG_UPSERT_*` substitution variables, and skips the upsert (no commit). Pair with `ASSERT` to halt explicitly:
+`PG_UPSERT`, `PG_UPSERT QA`, and `PG_UPSERT CHECK` do **not** raise an error when QA checks fail. The outcome is reported via `$PG_UPSERT_QA_PASSED` (and the per-table `$PG_UPSERT_TABLE_QA_PASSED`), so the script controls flow with `IF` or with execsql's standard error-halt configuration. When QA fails, the metacommand still runs `EXPORT_FAILURES`, populates all `$PG_UPSERT_*` substitution variables, and skips the upsert (no commit). To tell the ways a run can end apart, branch on `$PG_UPSERT_STATUS` rather than combining `$PG_UPSERT_QA_PASSED` with `$PG_UPSERT_COMMITTED`: a run with nothing to load (`no_changes`) and one the user cancelled (`cancelled`) both leave QA passed and nothing committed. Pair with `ASSERT` to halt explicitly:
 
 ```sql
 -- Soft failure: branch on the result, keep running.
@@ -2398,7 +2419,7 @@ Set after every `PG_UPSERT` execution:
 
 ### Database errors
 
-A database error during the run (a deadlock, a lock timeout, a constraint or trigger error the QA checks did not catch) is different from a QA failure. pg-upsert rolls the run back, and `PG_UPSERT` fails like any other metacommand: the driver's message is in `$ERROR_MESSAGE`, and the script halts unless `METACOMMAND_ERROR_HALT` is off. The `$PG_UPSERT_*` variables describe the failed run (`$PG_UPSERT_COMMITTED` is `FALSE` and the row counts are `0`), not an earlier `PG_UPSERT`.
+A database error during the run (a deadlock, a lock timeout, a constraint or trigger error the QA checks did not catch) is different from a QA failure. pg-upsert rolls the run back, and `PG_UPSERT` fails like any other metacommand: the driver's message is in `$ERROR_MESSAGE`, and the script halts unless `METACOMMAND_ERROR_HALT` is off. The `$PG_UPSERT_*` variables describe the failed run (`$PG_UPSERT_STATUS` is `db_error`, `$PG_UPSERT_ERROR` holds the message, `$PG_UPSERT_COMMITTED` is `FALSE` and the row counts are `0`), not an earlier `PG_UPSERT`. Cancelling at one of pg-upsert's own prompts is not an error: `PG_UPSERT` returns, with `$PG_UPSERT_STATUS` `cancelled`.
 
 The rollback ends the connection's transaction, so under `AUTOCOMMIT OFF` or inside `BEGIN BATCH`, statements run earlier on the same connection are rolled back with it.
 

{execsql2-2.32.0 → execsql2-2.34.0}/extras/vscode-execsql/package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "execsql-syntax",
-  "version": "2.32.0",
+  "version": "2.34.0",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "execsql-syntax",
-      "version": "2.32.0",
+      "version": "2.34.0",
       "license": "GPL-3.0-or-later",
       "dependencies": {
         "vscode-languageclient": "10.1.2"

{execsql2-2.32.0 → execsql2-2.34.0}/extras/vscode-execsql/package.json

@@ -2,7 +2,7 @@
   "name": "execsql-syntax",
   "displayName": "execsql",
   "description": "Syntax highlighting, lint findings, completion, hover, navigation, quick fixes and formatting for execsql scripts.",
-  "version": "2.32.0",
+  "version": "2.34.0",
   "publisher": "geocoug",
   "license": "GPL-3.0-or-later",
   "icon": "icon.png",

{execsql2-2.32.0 → execsql2-2.34.0}/extras/vscode-execsql/syntaxes/execsql.tmLanguage.json

@@ -118,7 +118,7 @@
     },
     "export-formats": {
       "comment": "Export/import format names",
-      "match": "(?i)\\b(json_tableschema|cgi-html|markdown|feather|json_ts|txt-and|unitsep|duckdb|sqlite|values|latex|plain|hdf5|html|json|tabq|tsvq|xlsx|yaml|b64|csv|ods|raw|tab|tsv|txt|xml|md|us)\\b",
+      "match": "(?i)\\b(json_tableschema|cgi-html|markdown|feather|json_ts|txt-and|unitsep|duckdb|sqlite|values|jsonl|latex|plain|hdf5|html|json|tabq|tsvq|xlsx|yaml|b64|csv|ods|raw|tab|tsv|txt|xml|md|us)\\b",
       "name": "support.constant.execsql"
     },
     "secondary-operators": {

{execsql2-2.32.0 → execsql2-2.34.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "execsql2"
-version = "2.32.0"
+version = "2.34.0"
 description = "Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables."
 readme = { file = "README.md", content-type = "text/markdown" }
 license = { file = "LICENSE.txt" }
@@ -188,7 +188,7 @@ line-ending = "auto"
 exclude = ["*.md"]
 
 [tool.bumpversion]
-current_version = "2.32.0"
+current_version = "2.34.0"
 commit = true
 tag = true
 tag_name = "v{new_version}"