execsql2 2.33.0__tar.gz → 2.34.0__tar.gz

{execsql2-2.33.0 → execsql2-2.34.0}/CHANGELOG.md

@@ -13,6 +13,26 @@ ______________________________________________________________________
 
 ______________________________________________________________________
 
+## [2.34.0] - 2026-10-18
+
+### Added
+
+- **`EXPORT … AS JSONL` and `EXPORT QUERY … AS JSONL`** write [JSON Lines](https://jsonlines.org/): one object per row, one row per line. `APPEND` adds rows to the file, which stays valid; `IMPORT … FROM JSON` reads it back.
+
+### Changed
+
+- **A value with a fraction loaded into an integer column** (`INTEGER`, `BIGINT`, `DECIMAL(10,0)`, …) of an existing table by `IMPORT` or `COPY` stops the load with an error that names the value and the column, and nothing is loaded. The database used to round it (`1.5` stored as `2`), or on SQL Server truncate it. A delimited file loaded into PostgreSQL or MySQL is not affected: the server reads it as written. Round the values in the source, or load them into a `DECIMAL` column with places for the fraction.
+- **MySQL / MariaDB sessions run in strict mode**: a SQL statement, `IMPORT` or `COPY` that gives a column a value MySQL can't store as given stops with MySQL's error, and nothing is loaded. MySQL used to store a changed value with a warning (`3000000000` as `2147483647`, `2024-02-30` as `0000-00-00`, `abcdef` in a `VARCHAR(3)` as `abc`). A timestamp ending in `Z` is one such value: write it with `+00:00`. To keep the old behavior in a script, run `SET SESSION sql_mode = 'ANSI';` after connecting.
+- **Scripts with many substitution variables run faster**: a statement or metacommand with no `!!`, `!'!` or `!"!` in it is no longer checked against every defined variable. With 500 variables each such line took about 0.1 ms longer; a 1,000-pass `LOOP` over `EXECUTE SCRIPT` with 200 variables now runs in half the time.
+
+### Fixed
+
+- **`IMPORT` and `COPY` into a table that already exists** convert each value by the type of the column it goes into, not the type guessed from the source data. A text column gets the value as written: `yes`, `1`, `01/05/2024`, `1e3` and `" 7 "` were stored as `true`, `true`, `2024-01-05 00:00:00`, `1000` and `7`. On MySQL a missing value in such a column is NULL, not `''`. `TO NEW` and `TO REPLACEMENT` tables are created and filled as before.
+- **`COPY` of a column of 0s and 1s into a PostgreSQL integer, numeric or float column** loads the numbers; it failed with "column … is of type integer but expression is of type boolean". The same goes for true/false values (`yes`/`no`, `t`/`f`) copied, or imported from a spreadsheet, JSON or Feather file, into such a column: they load as 1 and 0, as on the other databases.
+- **A missing date or timestamp loaded into MySQL by `IMPORT` or `COPY`** is stored as NULL; it was stored as `0000-00-00` (or `0` in a number column). Delimited files were not affected.
+
+______________________________________________________________________
+
 ## [2.33.0] - 2026-10-16
 
 ### Added

{execsql2-2.33.0 → execsql2-2.34.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: execsql2
-Version: 2.33.0
+Version: 2.34.0
 Summary: Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables.
 Project-URL: Homepage, https://execsql2.readthedocs.io/en/stable/
 Project-URL: Repository, https://github.com/geocoug/execsql
@@ -318,7 +318,7 @@ See the [full options reference](https://execsql2.readthedocs.io/en/stable/getti
 **Data movement**
 
 - Import data from CSV, TSV, JSON, Excel, OpenDocument, Feather, or Parquet files into a database table.
-- Export query results in 20+ formats including CSV, TSV, JSON, YAML, XML, HTML, Markdown, LaTeX, XLSX, OpenDocument, Feather, Parquet, HDF5, DuckDB, SQLite, plain text, and Jinja2 templates.
+- Export query results in 20+ formats including CSV, TSV, JSON, JSON Lines, YAML, XML, HTML, Markdown, LaTeX, XLSX, OpenDocument, Feather, Parquet, HDF5, DuckDB, SQLite, plain text, and Jinja2 templates.
 - Copy data between databases, including across different DBMS types.
 
 **Script logic**
@@ -441,7 +441,7 @@ execsql format --no-sql --in-place scripts/
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.33.0
+    rev: v2.34.0
     hooks:
       - id: execsql-format
       - id: execsql-lint

{execsql2-2.33.0 → execsql2-2.34.0}/README.md

@@ -198,7 +198,7 @@ See the [full options reference](https://execsql2.readthedocs.io/en/stable/getti
 **Data movement**
 
 - Import data from CSV, TSV, JSON, Excel, OpenDocument, Feather, or Parquet files into a database table.
-- Export query results in 20+ formats including CSV, TSV, JSON, YAML, XML, HTML, Markdown, LaTeX, XLSX, OpenDocument, Feather, Parquet, HDF5, DuckDB, SQLite, plain text, and Jinja2 templates.
+- Export query results in 20+ formats including CSV, TSV, JSON, JSON Lines, YAML, XML, HTML, Markdown, LaTeX, XLSX, OpenDocument, Feather, Parquet, HDF5, DuckDB, SQLite, plain text, and Jinja2 templates.
 - Copy data between databases, including across different DBMS types.
 
 **Script logic**
@@ -321,7 +321,7 @@ execsql format --no-sql --in-place scripts/
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.33.0
+    rev: v2.34.0
     hooks:
       - id: execsql-format
       - id: execsql-lint

{execsql2-2.33.0 → execsql2-2.34.0}/docs/about/divergence.md

@@ -43,6 +43,7 @@ ______________________________________________________________________
 | `PARQUET`         | Export query or table results to Apache Parquet via `polars`.                                                        |
 | `FEATHER`         | Export to Apache Feather/IPC via `polars` (upstream used `pandas` + `pyarrow`).                                      |
 | `YAML`            | Export query or table results as a YAML sequence of mappings via `PyYAML`.                                           |
+| `JSONL`           | Export query or table results as JSON Lines: one object per row, one row per line; `APPEND` adds rows (#7).          |
 | `MARKDOWN` / `MD` | Export query or table results as a GitHub-Flavored Markdown (GFM) pipe table. Pure Python, no optional dependencies. |
 | `XLSX`            | Export query or table results to an Excel XLSX workbook via `openpyxl` (single or multi-sheet).                      |
 
@@ -228,6 +229,12 @@ Upstream opened MySQL and MariaDB connections as `latin1`, and execsql2 did the
 
 The default is now `utf8mb4`. This does not change stored data or column charsets: the connection charset governs only how text moves between client and server, and MySQL transcodes to and from each column's own charset. A latin1 column read over a utf8mb4 connection returns the same string it always did. `-e latin1` restores the upstream behavior.
 
+### MySQL Strict Mode
+
+Upstream set `sql_mode='ANSI'` on every MySQL and MariaDB connection. That replaces the server's whole mode, and `ANSI` leaves out `STRICT_TRANS_TABLES`, so MySQL stored a changed value with a warning instead of raising an error: `3000000000` in an `INTEGER` column became `2147483647`, `2024-02-30` became `0000-00-00`, and `abcdef` in a `VARCHAR(3)` became `abc`. That applied to plain SQL statements, `IMPORT` and `COPY` alike (#101).
+
+The mode is now `ANSI,STRICT_TRANS_TABLES`, so such a value is an error, as in the `mysql` client with the server's default settings. The mode stays fixed rather than added to the server's own, because execsql's quoting of literals relies on backslash escapes being on (no `NO_BACKSLASH_ESCAPES`). `LOAD DATA LOCAL`, which `IMPORT` uses for a delimited file, ignores strict mode, so `IMPORT` checks its warnings and rolls the load back if MySQL changed a value. `SET SESSION sql_mode = 'ANSI';` in a script restores the upstream behavior for that session, `LOAD DATA` included.
+
 ### Database Support Tiers
 
 Upstream documented all nine DBMS adapters with equal confidence. execsql2 splits them by what is actually verified: **Supported** (PostgreSQL, MySQL/MariaDB, MS SQL Server, SQLite, DuckDB) run against a live server or a real database file on every CI run, while **best effort** (MS Access, Firebird, Oracle, ODBC DSN) are carried forward from the monolith and are exercised nowhere in CI.
@@ -436,6 +443,9 @@ These are behavioral changes driven by security or correctness issues in the ups
 | XLSX export wrote `numeric` columns as text                        | `_cell_value()` passed `int` and `float` straight to openpyxl but let every other type fall through to `str()`. PostgreSQL, MySQL, and DuckDB all return `Decimal` for `numeric`/`decimal` columns, so measurement values arrived in Excel as text cells — invisible in a screenshot, useless in a formula, a sort, or a chart. openpyxl lists `Decimal` in its own `NUMERIC_TYPES` and stores it as a real numeric cell, so the conversion was never needed. Found by exporting from a live PostgreSQL container and checking `cell.data_type` rather than `cell.value`.                                                                                                                                                                                                                                          |
 | `EMAIL` SMTP login without a password                              | Upstream calls `smtplib.login()` with only a username when `[email] password` and `enc_password` are both unset, which raises `TypeError` from the stdlib — `login()` has no single-argument form. execsql2 reports a configuration error naming the missing password instead.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
 | Type scan of large integers                                        | Working out a column's type tested each integer against the binary type by building `bytearray(n)`, n zero bytes, so a column of large integers (epoch milliseconds: about a terabyte per value) hung or ran out of memory. The binary test now answers for an integer without building it; the inferred types are unchanged.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+| Values loaded into an existing table                               | `IMPORT` and `COPY` converted every value to the type guessed from the source data, even when the table was already there with its own column types, so a text column got a different string than the source held (`yes` and `1` stored as `true`, `01/05/2024` as `2024-01-05 00:00:00`, `1e3` as `1000`), and on PostgreSQL a column of 0s and 1s copied into an integer column failed (the guess was boolean). For a table that exists, the target column's type now decides: a text column gets the value as written, a number column gets true/false values as 1 and 0, and other columns are converted as before. Each adapter reads the column types from its catalog (`pg_attribute`, `SHOW COLUMNS`, `sys.columns`) or its driver's description of the table. `TO NEW` / `REPLACEMENT` are unchanged.     |
+| Missing dates loaded into MySQL                                    | `IMPORT` and `COPY` sent a missing value in a column scanned as a date or timestamp as `''`, so MySQL stored `0000-00-00` (or `0` in a number column) instead of NULL. It is sent as NULL.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+| Fractions loaded into an integer column                            | A value with a fraction going into an integer column (including `DECIMAL(p,0)`, `NUMERIC(p)` and Oracle `NUMBER(p)`) of an existing table is an error naming the value and the column. Upstream sent the number and the database rounded it (`1.5` → `2`; SQL Server truncated it to `1`).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
 
 ______________________________________________________________________
 

{execsql2-2.33.0 → execsql2-2.34.0}/docs/api/db.md

@@ -8,6 +8,12 @@ If you are adding support for a new database, start with the [Adding Database Ad
 
 ::: execsql.db.base
 
+## Target column types
+
+`IMPORT` and `COPY` into a table that already exists convert each value by the type of the column it goes into (`Database.column_kinds()`), not by the type scanned from the source data.
+
+::: execsql.db.target_types
+
 ## Database factory
 
 Convenience constructors used internally to create typed `Database` instances. The CLI calls the appropriate factory function based on the `-t` flag value. Each factory validates its arguments (e.g., checking that a file exists) before constructing and returning the adapter.

{execsql2-2.33.0 → execsql2-2.34.0}/docs/dev/adding_db_adapters.md

@@ -133,23 +133,24 @@ class MyDBDatabase(Database):
 
 These are the instance attributes and methods you must configure correctly:
 
-| Attribute / Method | Type              | Purpose                                                                           |
-| ------------------ | ----------------- | --------------------------------------------------------------------------------- |
-| `self.type`        | `DbType`          | DBMS type token (e.g., `dbt_sqlite`). Controls quoting and type-mapping.          |
-| `self.paramstr`    | `str`             | SQL parameter placeholder: `"?"` (most drivers) or `"%s"` (psycopg, PyMySQL).     |
-| `self.encoding`    | `str`             | Database character encoding. Detect from the database on connect if possible.     |
-| `self.autocommit`  | `bool`            | `True` means the driver commits automatically; `False` requires explicit commits. |
-| `self.conn`        | driver connection | Set in `open_db()`.                                                               |
-| `open_db()`        | method            | **Must override.** Establish the connection and assign `self.conn`.               |
-| `table_exists()`   | method            | **Must override.** Query the DBMS catalog. Use parameterized queries.             |
-| `column_exists()`  | method            | **Must override.** Check column presence.                                         |
-| `table_columns()`  | method            | **Must override.** Return column names for a table.                               |
-| `view_exists()`    | method            | **Must override.** Check view presence.                                           |
-| `schema_exists()`  | method            | **Must override.** Return `False` if schemas are not supported.                   |
-| `drop_table()`     | method            | **Must override.** Drop a table (used by IMPORT when creating fresh).             |
-| `populate_table()` | method            | **Must override.** Bulk-load rows from a generator (used by IMPORT).              |
-| `row_inserter()`   | method            | Override for a faster bulk path that stores exactly what `executemany()` stores.  |
-| `exec_cmd()`       | method            | Override if the DBMS can execute stored procedures or views as commands.          |
+| Attribute / Method | Type              | Purpose                                                                                                                                                                             |
+| ------------------ | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `self.type`        | `DbType`          | DBMS type token (e.g., `dbt_sqlite`). Controls quoting and type-mapping.                                                                                                            |
+| `self.paramstr`    | `str`             | SQL parameter placeholder: `"?"` (most drivers) or `"%s"` (psycopg, PyMySQL).                                                                                                       |
+| `self.encoding`    | `str`             | Database character encoding. Detect from the database on connect if possible.                                                                                                       |
+| `self.autocommit`  | `bool`            | `True` means the driver commits automatically; `False` requires explicit commits.                                                                                                   |
+| `self.conn`        | driver connection | Set in `open_db()`.                                                                                                                                                                 |
+| `open_db()`        | method            | **Must override.** Establish the connection and assign `self.conn`.                                                                                                                 |
+| `table_exists()`   | method            | **Must override.** Query the DBMS catalog. Use parameterized queries.                                                                                                               |
+| `column_exists()`  | method            | **Must override.** Check column presence.                                                                                                                                           |
+| `table_columns()`  | method            | **Must override.** Return column names for a table.                                                                                                                                 |
+| `view_exists()`    | method            | **Must override.** Check view presence.                                                                                                                                             |
+| `schema_exists()`  | method            | **Must override.** Return `False` if schemas are not supported.                                                                                                                     |
+| `drop_table()`     | method            | **Must override.** Drop a table (used by IMPORT when creating fresh).                                                                                                               |
+| `populate_table()` | method            | **Must override.** Bulk-load rows from a generator (used by IMPORT).                                                                                                                |
+| `row_inserter()`   | method            | Override for a faster bulk path that stores exactly what `executemany()` stores.                                                                                                    |
+| `column_kinds()`   | method            | Override if the driver's description of a table does not tell its column types apart (see `db/target_types.py`); IMPORT and COPY into an existing table convert each value by them. |
+| `exec_cmd()`       | method            | Override if the DBMS can execute stored procedures or views as commands.                                                                                                            |
 
 Methods inherited from `Database` that you get for free include `execute()`, `cursor()`, `close()`, `commit()`, `rollback()`, `select_rowsource()`, `select_data()`, `schema_qualified_table_name()`, `quote_identifier()`, and `paramsubs()`.
 

{execsql2-2.33.0 → execsql2-2.34.0}/docs/dev/architecture.md

@@ -264,6 +264,8 @@ ______________________________________________________________________
 
 The base class provides shared implementations for `execute()`, `select_data()`, `select_rowsource()`, `select_rowdict()`, `commit()`, `rollback()`, and `populate_table()`.
 
+`populate_table()` converts each value to the type scanned from the source (`models.DataTable`). For a table that was already there (`existing_table=True`, passed by IMPORT and COPY), `column_kinds()` reports the target columns' types and `db/target_types.py` picks each column's conversion instead: a text column gets the value as given, an integer column whole numbers or an error. `column_kinds()` reads the driver's description of `select * from <table> where 1=0` by default; PostgreSQL, MySQL and SQL Server read their catalogs, where the description cannot tell some types apart. `tests/db/test_existing_table_loads.py` compares what every value type stores in every column type with the record in `tests/data/existing_table_loads/`.
+
 ### DatabasePool
 
 `DatabasePool` is a dict-like container mapping string aliases to open `Database` instances. It is the canonical `_state.dbs` object. Key operations:

{execsql2-2.33.0 → execsql2-2.34.0}/docs/guides/formatter.md

@@ -272,7 +272,7 @@ select id,name,created_at from users where active = true order by name;
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.33.0
+    rev: v2.34.0
     hooks:
       - id: execsql-format
       - id: execsql-lint # optional: fail the commit on lint errors

{execsql2-2.33.0 → execsql2-2.34.0}/docs/guides/sql_syntax.md

@@ -4,6 +4,26 @@
 
 When execsql connects to a SQL Server or MySQL/MariaDB database, it automatically configures the DBMS to expect ANSI-compatible SQL, to allow the use of more standards-compliant, and thus consistent, SQL. In particular, for MySQL/MariaDB, note that the double-quote character, rather than the backtick character, must be used to quote table, schema, and column names, and only the apostrophe can be used to quote character data.
 
+### MySQL strict mode { #mysql_strict_mode }
+
+On MySQL/MariaDB, execsql sets the session's `sql_mode` to `ANSI,STRICT_TRANS_TABLES`. A value that MySQL can't store as given is an error, for a SQL statement, [IMPORT](../reference/metacommands.md#import) or [COPY](../reference/metacommands.md#copy):
+
+```sql
+create table m (n integer, d date, s varchar(3));
+insert into m values (3000000000, '2024-02-30', 'abcdef');
+-- ERROR 1264 (22003): Out of range value for column 'n' at row 1
+```
+
+Without strict mode, MySQL would have stored `2147483647`, `0000-00-00` and `abc` with only a warning. This is how the `mysql` client behaves with the server's default settings. An IMPORT of a delimited file that MySQL reads with `LOAD DATA` is held to the same rule: if MySQL warns that it changed a value, the IMPORT stops and nothing is loaded.
+
+MySQL in strict mode does not read a trailing `Z` as a time zone: write `2024-01-05T10:00:00Z` as `2024-01-05 10:00:00+00:00`. A decimal with more places than its column has (`1.2345` into `DECIMAL(10,3)`) is still rounded, as MySQL does in strict mode.
+
+A script that relies on MySQL's changed values can turn strict mode off for the rest of its session:
+
+```sql
+SET SESSION sql_mode = 'ANSI';
+```
+
 ## Implicit Commits
 
 By default, execsql immediately commits all SQL statements. The [AUTOCOMMIT](../reference/metacommands.md#autocommit) metacommand can be used to turn off automatic commits, and the [BATCH](../reference/metacommands.md#batch) metacommand can be used to delay commits until the end of a batch. [IMPORT](../reference/metacommands.md#import) and [COPY](../reference/metacommands.md#copy) are the only metacommands that change data, and they also automatically commit their changes when complete (unless [AUTOCOMMIT](../reference/metacommands.md#autocommit) has been turned off). If a new table is created with either of these metacommands (through the use of the NEW or REPLACEMENT keywords), the CREATE TABLE statement will not be committed separately from the data addition, except when using Firebird. Thus, if an error occurs during addition of the data, the new target table will not exist---except when using Firebird.

{execsql2-2.33.0 → execsql2-2.34.0}/docs/index.md

@@ -9,7 +9,7 @@
 You can use *execsql* to:
 
 - Import data from text files and spreadsheets into a database.
-- Export tables and views to ~30 data formats including CSV, OpenDocument spreadsheets, XLSX, HTML, JSON, YAML, Markdown, LaTeX, XML, Parquet, Feather, HDF5, SQLite, and DuckDB.
+- Export tables and views to ~30 data formats including CSV, OpenDocument spreadsheets, XLSX, HTML, JSON, JSON Lines, YAML, Markdown, LaTeX, XML, Parquet, Feather, HDF5, SQLite, and DuckDB.
 - Copy data between different databases, even databases using different DBMSs.
 - Display tables or views on the terminal or in a GUI dialog window.
 - Export data using template processors to produce non-tabular output with customized format and contents.

{execsql2-2.33.0 → execsql2-2.34.0}/docs/reference/lint.md

@@ -158,7 +158,7 @@ The `execsql-lint` [pre-commit](https://pre-commit.com/) hook runs `execsql lint
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.33.0
+    rev: v2.34.0
     hooks:
       - id: execsql-lint
         args: [--ignore, V002] # optional: any execsql lint option

{execsql2-2.33.0 → execsql2-2.34.0}/docs/reference/metacommands.md

@@ -889,6 +889,8 @@ Copies the data from a data table or view in one database to a data table in a s
 
 The second (destination) table must have column names that are identical to the names of the columns in the first (source) table. The second table may have additional columns; if it does, they will not be affected and their names don't matter. The data types in the columns to be copied must be compatible, though not necessarily identical. The order of the columns in the two tables does not have to be identical.
 
+When neither "NEW" nor "REPLACEMENT" is used, each value is converted for the destination column it goes into, as for [IMPORT into an existing table](#import_existing_table_types): a text column gets the value as the source database returned it (an integer `1` is stored as `1`, a date as `2024-01-05`), an integer column gets whole numbers and stops the COPY with an error on a value with a fraction instead of rounding it, and a number column gets a column of 0s and 1s, or of true/false values, as 1 and 0.
+
 If the "NEW" keyword is used, the destination table will be automatically created with column names and data types that are compatible with the first (source) table. The data types used for the columns in the newly created table will be determined by a scan of all of the data in the first table, but may not exactly match those in the first table. A text column stays text, whatever its values look like: `00123`, `1`/`0` and `2024-01-05` are copied as written, not as numbers, booleans or dates. The source is read once; while its rows are scanned they are held in a temporary file, so a large copy does not need memory for the whole table. If the destination table already exists when the "NEW" keyword is used, the copy stops before creating anything, with the error `Table <name> already exists`; a name without a schema is looked for only in the schema the table would be created in (on MySQL, the current database)
 
 If the "REPLACEMENT" keyword is used, the destination table will also be created to be compatible with the source table, but any existing destination table of the same name will be dropped first. This replacement is not atomic: if the drop succeeds but a later create or load step fails, the old destination table has already been removed. If the drop fails, *execsql* stops the COPY command before creating or loading the replacement table. *execsql* uses a "drop table" statement to drop an existing destination table, and this statement may not succeed if there are dependencies on that table (see the discussion of [implicit drop table
@@ -1165,6 +1167,15 @@ JSON
 :   [Javascript Object Notation](http://www.json.org/). The data table is represented as an array of JSON objects, where each object represents a row of the table. Each row is represented as a set of key:value pairs, with column names used as the keys. No description text will be included in the output even if it is provided.
 
 
+JSONL
+
+:   [JSON Lines](https://jsonlines.org/): one JSON object per row, one row per line, with no surrounding array. Values are written as for JSON (dates and times as text, NULL as `null`); a line break inside a value is written as `\n`, so each row stays on one line. If the "APPEND" keyword is used, the rows are added to the end of the file, which stays valid JSON Lines (appending JSON instead leaves two arrays in one file). A query that returns no rows writes an empty file. The file can be read back with [IMPORT … FROM JSON](#import). No description text will be included in the output even if it is provided:
+
+    ```
+    -- !x! EXPORT QUERY <<select * from events where day = current_date;>> APPEND TO events.jsonl AS JSONL
+    ```
+
+
 JSON_TS or JSON_TABLESCHEMA
 
 :   [JSON Table Schema](https://frictionlessdata.io/specs/table-schema/). The data themselves are not exported. Ordinarily this form would be used in conjunction with an additional export of the data to a CSV file. The column type descriptions in the output are derived by scanning the entire table to evaluate the data type of each column. If a "NOTYPE" keyword is included before the "DESCRIPTION" keyword, this scanning is not done, and no type information is included in the output file.:
@@ -1945,7 +1956,7 @@ The syntax for importing data from a [JSON](https://www.json.org/) file is:
 IMPORT TO [NEW|REPLACEMENT] <table_name> FROM JSON <file_name>
 ```
 
-The JSON file must contain either a JSON array of objects (`[{…}, …]`) or [JSON Lines](https://jsonlines.org/) (JSONL, one object per line). Nested objects are flattened with dot-separated column names (e.g., an object `{"address": {"city": "Portland"}}` produces column `address.city`). Nested arrays within objects are stored as JSON strings. Records with different keys produce a superset of columns — missing keys become NULL. The parser auto-detects which form is present by peeking at the first non-whitespace character (`[` → array, `{` → JSON Lines); the file extension is not consulted.
+The JSON file must contain either a JSON array of objects (`[{…}, …]`) or [JSON Lines](https://jsonlines.org/) (JSONL, one object per line). Nested objects are flattened with dot-separated column names (e.g., an object `{"address": {"city": "Portland"}}` produces column `address.city`). Nested arrays within objects are stored as JSON strings. Records with different keys produce a superset of columns — missing keys become NULL. The parser auto-detects which form is present by peeking at the first non-whitespace character (`[` → array, `{` → JSON Lines); the file extension is not consulted. A file written by [EXPORT … AS JSONL](#export) is read as JSON Lines.
 
 Column names in the input must be valid for the DBMS in use.
 
@@ -1968,7 +1979,15 @@ A data line with fewer values than the header has NULL for the values left off t
 
 The rows are loaded in one transaction: if the load fails part-way (a constraint violation, a value of the wrong type, a row with too many columns), none of the file's rows are kept, on every DBMS. The exception is an ODBC DSN whose driver can only autocommit, which keeps the rows loaded before the failure; *execsql* warns once on such a connection.
 
-If neither the NEW or REPLACEMENT keywords are used, the table must exist, must have column names identical to those in the input data, and columns must have data types that are compatible with (though not necessarily identical to) those in the input data. If neither the NEW or REPLACEMENT keywords are used, the input data is not scanned to determine the data type of each column.
+<a id="import_existing_table_types"></a>
+If neither the NEW or REPLACEMENT keywords are used, the table must exist, must have column names identical to those in the input data, and columns must have data types that are compatible with (though not necessarily identical to) those in the input data. Each value is converted for the column it goes into:
+
+- A text column (`CHAR`, `VARCHAR`, `TEXT`, …) gets the value as written in the file: `yes`, `00701`, `01/05/2024` and `1e3` are stored as those strings. A missing value is NULL.
+- An integer column (`INTEGER`, `BIGINT`, `DECIMAL(10,0)`, `NUMERIC(10)`, Oracle `NUMBER(10)`, …) gets whole numbers: `7`, `+7`, `7.0` and `7e0` load as 7. A value with a fraction, such as `1.5`, stops the IMPORT with an error that names the value and the column, and nothing is loaded; it is not rounded.
+- A number column gets a column of true/false values (`yes`/`no`, `t`/`f`, `1`/`0`; see [BOOLEAN_INT](#boolean_int) and [BOOLEAN_WORDS](#boolean_words)) as 1 and 0.
+- Any other column (a date, a timestamp, a Boolean, a decimal with places for a fraction) gets the value converted to the data type found by scanning that column of the input, as with NEW, and the database converts it to the column's type.
+
+On PostgreSQL and MySQL a delimited text file goes to the server's own loader (`COPY`, `LOAD DATA`, or on PostgreSQL an `INSERT` of each value as text), and the server converts each value as written; the rules above apply to the other file formats, and to delimited files on the other DBMSs. On MySQL a value that the server can't store as given (out of range, an invalid date, text longer than the column) stops the IMPORT with an error, and nothing is loaded; see [MySQL strict mode](../guides/sql_syntax.md#mysql_strict_mode).
 
 If a table is scanned to determine data types, any column that is completely empty (all null) will be created with the text data type. This provides the greatest flexibility for subsequent addition of data to the table. However, if that column ought to have a different data type, and a WHERE clause is applied to that column assuming a different data type, the DBMS may report an error because of incomparable data types.
 
@@ -2032,7 +2051,7 @@ When MS-Excel saves an OpenDocument spreadsheet, it may create an additional emp
 Because the data addition to the target table is always committed except when [AUTOCOMMIT](#autocommit) is off, the IMPORT metacommand should be used with care within transactions that are managed with explicit SQL statements or with [BATCHes](#batch).
 
 
-In general, if an error occurs while importing data, none of the new data should be in the target table (the operation is not committed). However, MySQL/MariaDB may issue messages about data type incompatibility to the standard error device (ordinarily the terminal), yet load some or all of the data. If the NEW or REPLACEMENT keywords are used, depending on the DBMS and where the error occurred, the target table may be created even if the data are not loaded.
+In general, if an error occurs while importing data, none of the new data should be in the target table (the operation is not committed). On MySQL/MariaDB a value that the server would change to store it is an error too, unless the script has turned off [strict mode](../guides/sql_syntax.md#mysql_strict_mode). If the NEW or REPLACEMENT keywords are used, depending on the DBMS and where the error occurred, the target table may be created even if the data are not loaded.
 
 The name, size, and date of the IMPORTed file are written to the [execsql.log](../guides/logging.md#logging) file.
 

{execsql2-2.33.0 → execsql2-2.34.0}/extras/vscode-execsql/package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "execsql-syntax",
-  "version": "2.33.0",
+  "version": "2.34.0",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "execsql-syntax",
-      "version": "2.33.0",
+      "version": "2.34.0",
       "license": "GPL-3.0-or-later",
       "dependencies": {
         "vscode-languageclient": "10.1.2"

{execsql2-2.33.0 → execsql2-2.34.0}/extras/vscode-execsql/package.json

@@ -2,7 +2,7 @@
   "name": "execsql-syntax",
   "displayName": "execsql",
   "description": "Syntax highlighting, lint findings, completion, hover, navigation, quick fixes and formatting for execsql scripts.",
-  "version": "2.33.0",
+  "version": "2.34.0",
   "publisher": "geocoug",
   "license": "GPL-3.0-or-later",
   "icon": "icon.png",

{execsql2-2.33.0 → execsql2-2.34.0}/extras/vscode-execsql/syntaxes/execsql.tmLanguage.json

@@ -118,7 +118,7 @@
     },
     "export-formats": {
       "comment": "Export/import format names",
-      "match": "(?i)\\b(json_tableschema|cgi-html|markdown|feather|json_ts|txt-and|unitsep|duckdb|sqlite|values|latex|plain|hdf5|html|json|tabq|tsvq|xlsx|yaml|b64|csv|ods|raw|tab|tsv|txt|xml|md|us)\\b",
+      "match": "(?i)\\b(json_tableschema|cgi-html|markdown|feather|json_ts|txt-and|unitsep|duckdb|sqlite|values|jsonl|latex|plain|hdf5|html|json|tabq|tsvq|xlsx|yaml|b64|csv|ods|raw|tab|tsv|txt|xml|md|us)\\b",
       "name": "support.constant.execsql"
     },
     "secondary-operators": {

{execsql2-2.33.0 → execsql2-2.34.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "execsql2"
-version = "2.33.0"
+version = "2.34.0"
 description = "Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables."
 readme = { file = "README.md", content-type = "text/markdown" }
 license = { file = "LICENSE.txt" }
@@ -188,7 +188,7 @@ line-ending = "auto"
 exclude = ["*.md"]
 
 [tool.bumpversion]
-current_version = "2.33.0"
+current_version = "2.34.0"
 commit = true
 tag = true
 tag_name = "v{new_version}"

{execsql2-2.33.0 → execsql2-2.34.0}/src/execsql/db/base.py

@@ -24,7 +24,9 @@ from collections.abc import Callable, Generator, Iterator
 
 from execsql.exceptions import ErrInfo
 from execsql.utils.errors import exception_desc
+from execsql.db.target_types import ColumnKinds, existing_table_converter, kind_for_python_type
 from execsql.db.tiers import SupportTier, announce_tier
+from execsql.types import DataType
 import execsql.state as _state
 
 __all__ = ["Database", "DatabasePool"]
@@ -604,6 +606,42 @@ class Database(ABC):
             rows = curs.fetchall()
         return [row[0] for row in rows]
 
+    def column_kinds(self, sq_name: str) -> ColumnKinds:
+        """The execsql data type each column of table *sq_name* converts values to (see ``db.target_types``).
+
+        *sq_name* is schema-qualified and quoted, as for an ``INSERT``, and
+        is looked up the way the ``INSERT`` finds it.  Keys are lower-cased
+        column names; a column with no fitting execsql type maps to ``None``.
+        This default reads the driver's description of an empty query on the
+        table; adapters whose catalog names types more exactly override it.
+        """
+        description, _rows = self._column_type_query(sq_name, f"select * from {sq_name} where 1=0;")
+        return {d[0].lower(): self._described_kind(d) for d in description}
+
+    def _described_kind(self, description: Any) -> type[DataType] | None:
+        """The execsql type for one column of a cursor description (a DB-API 7-item sequence)."""
+        return kind_for_python_type(description[1], description[5])
+
+    def _column_type_query(self, sq_name: str, sql: str, params: list | None = None) -> tuple[Any, list]:
+        """Run *sql* for :meth:`column_kinds` of *sq_name*; return the cursor description and the rows."""
+        with self._cursor() as curs:
+            try:
+                if params is None:
+                    curs.execute(sql)
+                else:
+                    curs.execute(sql, params)
+            except ErrInfo:
+                raise
+            except Exception as e:
+                self.rollback()
+                raise ErrInfo(
+                    type="db",
+                    command_text=sql,
+                    exception_msg=exception_desc(),
+                    other_msg=f"Failed to get the column types of table {sq_name} of {self.name()}",
+                ) from e
+            return curs.description, curs.fetchall()
+
     def view_exists(self, view_name: str, schema_name: str | None = None) -> bool:
         """Return ``True`` if *view_name* (optionally in *schema_name*) exists."""
         with self._cursor() as curs:
@@ -689,11 +727,17 @@ class Database(ABC):
         rowsource: Any,
         column_list: list[str],
         tablespec_src: Callable,
+        *,
+        existing_table: bool = False,
     ) -> None:
         """Bulk-insert rows from *rowsource* into *table_name* using the columns in *column_list*.
 
         *rowsource* must be a generator yielding lists of values in column order.
         *tablespec_src* is a zero-argument callable that returns the table's type specification.
+        Each value is converted to the type scanned for its column; with
+        *existing_table* (the table was there before the IMPORT or COPY), the
+        type of the column it goes into decides instead (see
+        ``db.target_types.existing_table_converter``).
         """
         # The rowsource argument must be a generator yielding a list of values for the columns of the table.
         # The column_list argument must an iterable containing column names.  This may be a subset of
@@ -713,8 +757,13 @@ class Database(ABC):
         sel_cols = [col for col in ts_colnames if col in column_list]
         incl_col = [col in column_list for col in ts_colnames]
         # Type conversion functions for the rowsource.
-        type_objs = [col.column_type()[1]() for col in tablespec.cols]
-        type_mod_fn = [self.type.dialect[col.column_type()[1]][3] for col in tablespec.cols]
+        converters = [self._scanned_type_converter(col.column_type()[1]) for col in tablespec.cols]
+        if existing_table:
+            kinds = self.column_kinds(sq_name)
+            converters = [
+                existing_table_converter(kinds.get(col.name.lower()), col.column_type()[1], convert, col.name, sq_name)
+                for col, convert in zip(tablespec.cols, converters)
+            ]
         # Construct INSERT statement.
         columns = [self.type.quoted(col) for col in sel_cols]
         colspec = ",".join(columns)
@@ -812,10 +861,7 @@ class Database(ABC):
                                             line[i] = None
                             # Trailing values left off the line are NULL.
                             line = list(line) + [None] * (len(ts_colnames) - len(line))
-                            lt = [
-                                type_objs[i].from_data(val) if val is not None else None for i, val in enumerate(line)
-                            ]
-                            lt = [type_mod_fn[i](v) if type_mod_fn[i] else v for i, v in enumerate(lt)]
+                            lt = [converters[i](val) for i, val in enumerate(line)]
                             row = []
                             for i, v in enumerate(lt):
                                 if incl_col[i]:
@@ -869,14 +915,31 @@ class Database(ABC):
                 f"IMPORT into {sq_name} complete: {total_rows} rows imported.",
             )
 
+    def _scanned_type_converter(self, data_type: type[DataType]) -> Callable[[Any], Any]:
+        """Convert a value to *data_type*, then apply this DBMS's adjustment for that type, if any.
+
+        NULL is not converted, but it is adjusted: MySQL's adjustment for
+        dates turns it into ``""``.
+        """
+        from_data = data_type().from_data
+        adjust = self.type.dialect[data_type][3]
+        if adjust is None:
+            return lambda value: None if value is None else from_data(value)
+        return lambda value: adjust(None if value is None else from_data(value))
+
     def import_tabular_file(
         self,
         schema_name: str | None,
         table_name: str,
         csv_file_obj: Any,
         skipheader: bool,
+        *,
+        existing_table: bool = False,
     ) -> None:
-        """Import a CSV/tabular file into *table_name*; column names must be compatible."""
+        """Import a CSV/tabular file into *table_name*; column names must be compatible.
+
+        *existing_table*: the table was there before the IMPORT (see :meth:`populate_table`).
+        """
         # Import a text (CSV) file containing tabular data to a table.  Columns must be compatible.
         if not self.table_exists(table_name, schema_name):
             raise ErrInfo(
@@ -912,7 +975,7 @@ class Database(ABC):
 
         f = csv_file_obj.reader()
         next(f)
-        self.populate_table(schema_name, table_name, f, import_cols, get_ts)
+        self.populate_table(schema_name, table_name, f, import_cols, get_ts, existing_table=existing_table)
 
     def import_entire_file(
         self,

{execsql2-2.33.0 → execsql2-2.34.0}/src/execsql/db/duckdb.py

@@ -18,6 +18,8 @@ from pathlib import Path
 from typing import Any
 
 from execsql.db.base import Database, statement_start
+from execsql.db.target_types import kind_for_type_name
+from execsql.types import DataType
 from execsql.exceptions import ErrInfo
 from execsql.utils.errors import exception_desc, fatal_error, install_hint
 import execsql.state as _state
@@ -320,6 +322,10 @@ class DuckDBDatabase(Database):
 
         return insert
 
+    def _described_kind(self, description: Any) -> type[DataType] | None:
+        """DuckDB describes a column by its type's name (``DECIMAL(18,3)``, ``TIMESTAMP WITH TIME ZONE``)."""
+        return kind_for_type_name(str(description[1]))
+
     def view_exists(self, view_name: str, schema_name: str | None = None) -> bool:
         """Return True if the named view exists in the DuckDB database."""
         # DuckDB information_schema has no 'views' table; views are listed in 'tables'