execsql2 2.31.1__tar.gz → 2.32.0__tar.gz

{execsql2-2.31.1 → execsql2-2.32.0}/CHANGELOG.md

@@ -13,6 +13,27 @@ ______________________________________________________________________
 
 ______________________________________________________________________
 
+## [2.32.0] - 2026-10-16
+
+### Changed
+
+- **Working out column types is about twice as fast** for `IMPORT`, `COPY` and the exports that create a table (`EXPORT … AS SQLITE`, `DUCKDB`, `JSON`, `FEATHER`); a CSV whose dates repeat scans close to three times as fast. The types chosen are the same.
+- **`IMPORT`, `COPY` and `EXPORT … AS DUCKDB` into DuckDB are about five times as fast**: they insert many rows per statement. Copying 200,000 rows took 124 s and now takes 22 s.
+
+### Fixed
+
+- **A column of large integers**, such as epoch milliseconds, no longer makes `IMPORT`, `COPY` or those exports hang or run out of memory while execsql works out the column types.
+
+______________________________________________________________________
+
+## [2.31.2] - 2026-10-16
+
+### Fixed
+
+- **SQL Server, Access and ODBC DSN connections** start a new database session on every `CONNECT` and `execsql.run()`: temp tables and `SET` options no longer carry over from an earlier connection in the same process on macOS and Linux. execsql turns off pyodbc's connection pooling, which also applies to other pyodbc connections your program makes.
+
+______________________________________________________________________
+
 ## [2.31.1] - 2026-10-16
 
 ### Added

{execsql2-2.31.1 → execsql2-2.32.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: execsql2
-Version: 2.31.1
+Version: 2.32.0
 Summary: Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables.
 Project-URL: Homepage, https://execsql2.readthedocs.io/en/stable/
 Project-URL: Repository, https://github.com/geocoug/execsql
@@ -441,7 +441,7 @@ execsql format --no-sql --in-place scripts/
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.31.1
+    rev: v2.32.0
     hooks:
       - id: execsql-format
       - id: execsql-lint

{execsql2-2.31.1 → execsql2-2.32.0}/README.md

@@ -321,7 +321,7 @@ execsql format --no-sql --in-place scripts/
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.31.1
+    rev: v2.32.0
     hooks:
       - id: execsql-format
       - id: execsql-lint

{execsql2-2.31.1 → execsql2-2.32.0}/docs/about/divergence.md

@@ -264,6 +264,8 @@ All 33 mutable runtime globals in `state.py` have been consolidated into a `Runt
 - **PostgreSQL driver is psycopg3** — the PostgreSQL adapter uses `psycopg` (psycopg3, installed via the `psycopg[binary]` extra) instead of upstream's `psycopg2`. The connection now passes the libpq `dbname` keyword (psycopg3 rejects psycopg2's `database`), `VACUUM` toggles `conn.autocommit` instead of the removed `set_session()`, server-side `COPY` import uses psycopg3's `cursor.copy()` context manager instead of `copy_expert()`, and binary `IMPORT` sends raw `bytes` instead of `psycopg2.Binary()`. This also makes `db.conn` directly usable by the `PG_UPSERT` metacommand's pg-upsert ≥1.26.0 dependency, which itself requires a psycopg3 connection. The connection sets `prepare_threshold=None` to disable psycopg3's automatic server-side prepared statements, which psycopg2 never used; without this, re-running a query (e.g. via `EXPORT`) against an object the script drops/recreates or alters raises `FeatureNotSupported: cached plan must not change result type`.
 - **Oracle driver is python-oracledb** — the Oracle adapter imports `oracledb`, the driver the `oracle` extra installs, instead of upstream's `cx_Oracle` (which ships no wheels for Python 3.12+). The `makedsn` / `connect` calls are the same. `cx_Oracle` is still used when it is the only Oracle driver installed.
 - **SQL Server tries ODBC Driver 18 first** — upstream's driver list stopped at Driver 17, so a machine with only Microsoft's current driver could not connect. Driver 18 encrypts the connection and verifies the server's certificate; the `trust_server_certificate` setting accepts a self-signed one.
+- **DuckDB bulk inserts** — `IMPORT`, `COPY` and `EXPORT ... AS DUCKDB` insert many rows per `INSERT ... VALUES` statement instead of `executemany()`, which DuckDB runs one row at a time; the values stored are the same. While binding them, and only when pandas is not installed, `sys.modules["pandas"]` is `None`: the duckdb package looks pandas up twice per row, and without that entry each lookup searched `sys.path`.
+- **Each pyodbc connection is a new session** — the SQL Server, Access and ODBC DSN adapters set `pyodbc.pooling = False` before they connect. With pyodbc's default pooling, unixODBC (macOS, Linux) handed a closed connection's session unchanged to the next connection to the same server, so temp tables and `SET` options from one `CONNECT`, or one `execsql.run()`, were still there in the next. pyodbc applies the setting at the process's first pyodbc connection, so it also covers other pyodbc connections the process makes, and has no effect if the program connected with pyodbc before execsql did.
 - **`Database` is an ABC** — `open_db()` and `exec_cmd()` are abstract methods. Subclasses that omit them raise `TypeError` at instantiation instead of at call time.
 - **Connection timeouts** — PostgreSQL and SQLite adapters accept a connection timeout parameter (default 30 seconds).
 - **DuckDB temporal types** — `TIMESTAMPTZ`, `TIMESTAMP`, `DATE`, `TIME` now map to native DuckDB types instead of `TEXT`.
@@ -433,6 +435,7 @@ These are behavioral changes driven by security or correctness issues in the ups
 | ODS export replaced newlines inside a value with spaces            | String cells were written after `item.replace("\n", " ")`, so an exported value silently differed from the one in the database. An XML parser does normalise a newline inside an attribute, so `office:string-value` cannot carry one — but ODF represents a multi-line cell as one `<text:p>` per line, which is what LibreOffice writes and what execsql2 now writes. Single-line values still use the attribute.                                                                                                                                                                                                                                                                                                                                                                                                |
 | XLSX export wrote `numeric` columns as text                        | `_cell_value()` passed `int` and `float` straight to openpyxl but let every other type fall through to `str()`. PostgreSQL, MySQL, and DuckDB all return `Decimal` for `numeric`/`decimal` columns, so measurement values arrived in Excel as text cells — invisible in a screenshot, useless in a formula, a sort, or a chart. openpyxl lists `Decimal` in its own `NUMERIC_TYPES` and stores it as a real numeric cell, so the conversion was never needed. Found by exporting from a live PostgreSQL container and checking `cell.data_type` rather than `cell.value`.                                                                                                                                                                                                                                          |
 | `EMAIL` SMTP login without a password                              | Upstream calls `smtplib.login()` with only a username when `[email] password` and `enc_password` are both unset, which raises `TypeError` from the stdlib — `login()` has no single-argument form. execsql2 reports a configuration error naming the missing password instead.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
+| Type scan of large integers                                        | Working out a column's type tested each integer against the binary type by building `bytearray(n)`, n zero bytes, so a column of large integers (epoch milliseconds: about a terabyte per value) hung or ran out of memory. The binary test now answers for an integer without building it; the inferred types are unchanged.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
 
 ______________________________________________________________________
 

{execsql2-2.31.1 → execsql2-2.32.0}/docs/api/index.md

@@ -78,11 +78,12 @@ for t in threads:
 
 Each run has its own [file writer](#file-output), so two runs writing files at the same time both get complete files, and a run that halts on an error stops only its own output. Plugin metacommands are registered once per process, not once per call, and a `--manifest` recorder belongs to the run that started it.
 
-Three things are still shared by every thread in the process, so keep them apart yourself:
+Four things are still shared by every thread in the process, so keep them apart yourself:
 
 - **The working directory.** `CD` calls `os.chdir()`, which moves every thread while the script runs. Use absolute paths in scripts that run concurrently, or don't use `CD` in them. When a script that ran `CD` finishes, `run()` changes back to the directory it was in before that `CD`, so the caller ends where it started; a run without `CD` leaves the directory as it finds it.
 - **The same output file.** Two runs writing one file at the same time can interleave their lines. Give each run its own output paths.
 - **Interactive prompts.** `PROMPT` and the other GUI metacommands are not meant to be driven from several threads at once.
+- **pyodbc's connection pooling.** Before its first SQL Server, Access or ODBC DSN connection, execsql turns pooling off (`pyodbc.pooling = False`), so every `CONNECT` and every `run()` gets a new database session: a temp table or `SET` option from one run is gone in the next. pyodbc reads the setting at the process's first pyodbc connection, so your own pyodbc connections are not pooled either, and if your program connects with pyodbc before execsql does, pooling stays on and runs against the same server can share a session.
 
 ### File Output { #file-output }
 

{execsql2-2.31.1 → execsql2-2.32.0}/docs/dev/adding_db_adapters.md

@@ -148,6 +148,7 @@ These are the instance attributes and methods you must configure correctly:
 | `schema_exists()`  | method            | **Must override.** Return `False` if schemas are not supported.                   |
 | `drop_table()`     | method            | **Must override.** Drop a table (used by IMPORT when creating fresh).             |
 | `populate_table()` | method            | **Must override.** Bulk-load rows from a generator (used by IMPORT).              |
+| `row_inserter()`   | method            | Override for a faster bulk path that stores exactly what `executemany()` stores.  |
 | `exec_cmd()`       | method            | Override if the DBMS can execute stored procedures or views as commands.          |
 
 Methods inherited from `Database` that you get for free include `execute()`, `cursor()`, `close()`, `commit()`, `rollback()`, `select_rowsource()`, `select_data()`, `schema_qualified_table_name()`, `quote_identifier()`, and `paramsubs()`.

{execsql2-2.31.1 → execsql2-2.32.0}/docs/dev/architecture.md

@@ -297,7 +297,7 @@ ______________________________________________________________________
 1. An `IMPORT` metacommand is dispatched to the appropriate handler.
 1. The handler instantiates the correct importer (e.g., `importers/csv.py`, `importers/ods.py`) based on the file type.
 1. The importer reads the file and yields rows.
-1. `Database.populate_table()` bulk-inserts the rows using `executemany()`, with optional progress bars.
+1. `Database.populate_table()` converts the rows and inserts them in batches of `import_row_buffer` rows, in one transaction, with optional progress bars. Each batch goes through the adapter's `row_inserter()`: `executemany()` by default, and multi-row `INSERT ... VALUES` on DuckDB, whose `executemany()` binds one row at a time.
 
 ### Export metadata
 

{execsql2-2.31.1 → execsql2-2.32.0}/docs/guides/formatter.md

@@ -272,7 +272,7 @@ select id,name,created_at from users where active = true order by name;
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.31.1
+    rev: v2.32.0
     hooks:
       - id: execsql-format
       - id: execsql-lint # optional: fail the commit on lint errors

{execsql2-2.31.1 → execsql2-2.32.0}/docs/reference/lint.md

@@ -158,7 +158,7 @@ The `execsql-lint` [pre-commit](https://pre-commit.com/) hook runs `execsql lint
 ```yaml
 repos:
   - repo: https://github.com/geocoug/execsql
-    rev: v2.31.1
+    rev: v2.32.0
     hooks:
       - id: execsql-lint
         args: [--ignore, V002] # optional: any execsql lint option

{execsql2-2.31.1 → execsql2-2.32.0}/extras/vscode-execsql/package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "execsql-syntax",
-  "version": "2.31.1",
+  "version": "2.32.0",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "execsql-syntax",
-      "version": "2.31.1",
+      "version": "2.32.0",
       "license": "GPL-3.0-or-later",
       "dependencies": {
         "vscode-languageclient": "10.1.2"

{execsql2-2.31.1 → execsql2-2.32.0}/extras/vscode-execsql/package.json

@@ -2,7 +2,7 @@
   "name": "execsql-syntax",
   "displayName": "execsql",
   "description": "Syntax highlighting, lint findings, completion, hover, navigation, quick fixes and formatting for execsql scripts.",
-  "version": "2.31.1",
+  "version": "2.32.0",
   "publisher": "geocoug",
   "license": "GPL-3.0-or-later",
   "icon": "icon.png",

{execsql2-2.31.1 → execsql2-2.32.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "execsql2"
-version = "2.31.1"
+version = "2.32.0"
 description = "Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables."
 readme = { file = "README.md", content-type = "text/markdown" }
 license = { file = "LICENSE.txt" }
@@ -188,7 +188,7 @@ line-ending = "auto"
 exclude = ["*.md"]
 
 [tool.bumpversion]
-current_version = "2.31.1"
+current_version = "2.32.0"
 commit = true
 tag = true
 tag_name = "v{new_version}"

{execsql2-2.31.1 → execsql2-2.32.0}/src/execsql/db/access.py

@@ -15,7 +15,7 @@ from pathlib import Path
 from typing import Any, cast
 
 from execsql.db.base import Database, _fetched
-from execsql.db.odbc import odbc_value
+from execsql.db.odbc import fresh_sessions, odbc_value
 from execsql.db.tiers import SupportTier
 from execsql.exceptions import ErrInfo
 from execsql.utils.errors import exception_desc, fatal_error, install_hint
@@ -108,6 +108,8 @@ class AccessDatabase(Database):
         # Open an ODBC connection.
         import pyodbc
 
+        fresh_sessions(pyodbc)
+
         if self.conn is not None:
             self.conn.close()
             self.conn = None

{execsql2-2.31.1 → execsql2-2.32.0}/src/execsql/db/base.py

@@ -665,6 +665,23 @@ class Database(ABC):
         self.execute(f"drop table if exists {tablename} cascade;")
         self.commit()
 
+    def row_inserter(self, curs: Any, table: str, columns: list[str]) -> Callable[[list[list[Any]]], None]:
+        """Return a function that inserts one batch of rows into *table* through *curs*.
+
+        *table* is schema-qualified and quoted; *columns* are the quoted
+        column names, in the order of each row's values.  IMPORT and COPY call
+        the function once per batch, all inside one transaction.  This one
+        runs ``executemany``.  An adapter may override it with a faster bulk
+        path only if that path stores exactly what ``executemany`` stores, for
+        every value type and column type: a faster load must not change data.
+        """
+        sql = f"insert into {table} ({','.join(columns)}) values ({self.paramsubs(len(columns))});"
+
+        def insert(rows: list[list[Any]]) -> None:
+            curs.executemany(sql, rows)
+
+        return insert
+
     def populate_table(
         self,
         schema_name: str | None,
@@ -740,6 +757,7 @@ class Database(ABC):
 
         def _import_loop(curs) -> int:
             nonlocal eof, total_rows, task_id
+            insert = self.row_inserter(curs, sq_name, columns)
             while True:
                 b = []
                 for _j in range(_state.conf.import_row_buffer):
@@ -809,7 +827,7 @@ class Database(ABC):
                                 b.append(row)
                 if len(b) > 0:
                     try:
-                        curs.executemany(sql, b)
+                        insert(b)
                     except ErrInfo:
                         raise
                     except Exception as e:

{execsql2-2.31.1 → execsql2-2.32.0}/src/execsql/db/dsn.py

@@ -10,6 +10,7 @@ the CLI.
 
 
 from execsql.db.base import Database
+from execsql.db.odbc import fresh_sessions
 from execsql.db.tiers import SupportTier
 from execsql.exceptions import ErrInfo
 from execsql.utils.errors import exception_desc, fatal_error, install_hint
@@ -75,6 +76,8 @@ class DsnDatabase(Database):
         # Open an ODBC connection using a DSN.
         import pyodbc
 
+        fresh_sessions(pyodbc)
+
         if self.conn is not None:
             self.conn.close()
             self.conn = None

{execsql2-2.31.1 → execsql2-2.32.0}/src/execsql/db/duckdb.py

@@ -8,8 +8,12 @@ analytics databases via the ``duckdb`` package.  Corresponds to ``-t k``
 on the CLI.
 """
 
+import importlib.util
 import re
+import sys
 from collections import deque
+from collections.abc import Callable, Iterator
+from contextlib import contextmanager
 from pathlib import Path
 from typing import Any
 
@@ -18,7 +22,7 @@ from execsql.exceptions import ErrInfo
 from execsql.utils.errors import exception_desc, fatal_error, install_hint
 import execsql.state as _state
 
-__all__ = ["DuckDBDatabase"]
+__all__ = ["DuckDBDatabase", "insert_rows"]
 
 # Statements that return rows; any other statement DuckDB answers with a
 # one-column "Count" result (rows inserted, updated or deleted).
@@ -28,6 +32,53 @@ _QUERY_RX = re.compile(r"(?:SELECT|WITH|VALUES|FROM|TABLE|SHOW|DESCRIBE|SUMMARIZ
 _TRANSACTION_RX = re.compile(r"(?P<begin>BEGIN|START\s+TRANSACTION)\b|(?P<end>COMMIT|END|ROLLBACK|ABORT)\b", re.I)
 
 
+# Parameters bound by one INSERT ... VALUES statement in insert_rows(); a
+# wide table gets fewer rows per statement.
+_PARAMS_PER_STATEMENT = 10_000
+
+
+@contextmanager
+def _pandas_lookup_skipped() -> Iterator[None]:
+    """Make DuckDB's check for pandas fail at once while it binds rows.
+
+    The duckdb package asks Python's import system for ``pandas`` twice for
+    every row it binds, and when pandas is not installed nothing remembers
+    the answer, so each row searches ``sys.path`` again: about three
+    quarters of an INSERT's time.  A ``None`` in ``sys.modules`` makes that
+    import fail without the search.  It is set only when pandas is not
+    installed, so no import that could have succeeded fails, and it is
+    removed afterwards.
+    """
+    if "pandas" in sys.modules or importlib.util.find_spec("pandas") is not None:
+        yield
+        return
+    sys.modules["pandas"] = None  # type: ignore[assignment]
+    try:
+        yield
+    finally:
+        if "pandas" in sys.modules and sys.modules["pandas"] is None:
+            del sys.modules["pandas"]
+
+
+def insert_rows(curs: Any, table: str, columns: list[str], rows: list[Any]) -> None:
+    """Insert *rows* into *table* with multi-row ``INSERT ... VALUES`` statements.
+
+    *table* is quoted (and schema-qualified as needed); *columns* are the
+    quoted column names, in the order of each row's values.  DuckDB's
+    ``executemany`` runs one row at a time; binding many rows in one
+    statement is several times faster and stores the same values.
+    """
+    if not rows:
+        return
+    per_statement = max(1, _PARAMS_PER_STATEMENT // max(1, len(columns)))
+    head = f"insert into {table} ({','.join(columns)}) values "
+    row_marks = "(" + ",".join("?" * len(columns)) + ")"
+    with _pandas_lookup_skipped():
+        for start in range(0, len(rows), per_statement):
+            chunk = rows[start : start + per_statement]
+            curs.execute(head + ",".join([row_marks] * len(chunk)), [value for row in chunk for value in row])
+
+
 class _SharedCursor:
     """A cursor that runs on the database's one connection.
 
@@ -261,6 +312,14 @@ class DuckDBDatabase(Database):
             curs.execute(sql, params)
             return len(curs.fetchall()) > 0
 
+    def row_inserter(self, curs: Any, table: str, columns: list[str]) -> Callable[[list[list[Any]]], None]:
+        """Insert each batch with multi-row ``INSERT ... VALUES`` (:func:`insert_rows`)."""
+
+        def insert(rows: list[list[Any]]) -> None:
+            insert_rows(curs, table, columns, rows)
+
+        return insert
+
     def view_exists(self, view_name: str, schema_name: str | None = None) -> bool:
         """Return True if the named view exists in the DuckDB database."""
         # DuckDB information_schema has no 'views' table; views are listed in 'tables'

{execsql2-2.31.1 → execsql2-2.32.0}/src/execsql/db/odbc.py

@@ -7,15 +7,32 @@ taken from a substitution variable) would end its attribute early and
 start another.  :func:`odbc_value` brace-quotes such values;
 :func:`connection_string` joins the attributes, and :func:`loggable`
 gives the same string with the password masked for the log.
+:func:`fresh_sessions` makes each pyodbc connection a new database session.
 """
 
 from __future__ import annotations
 
-__all__ = ["connection_string", "loggable", "odbc_value"]
+from typing import Any
+
+__all__ = ["connection_string", "fresh_sessions", "loggable", "odbc_value"]
 
 _SECRET_KEYS = frozenset({"pwd", "password"})
 
 
+def fresh_sessions(pyodbc: Any) -> None:
+    """Turn off pyodbc's connection pooling, so each connection is a new session.
+
+    With pooling on, closing a connection hands its session back to the ODBC
+    driver manager, and unixODBC gives it unchanged to the next connection to
+    the same server: temp tables and ``SET`` options from one ``CONNECT`` or
+    ``execsql.run()`` were still there in the next.  pyodbc applies the
+    setting when the process makes its first pyodbc connection, so it covers
+    every pyodbc connection in the process, and has no effect if one was
+    made before execsql's first.
+    """
+    pyodbc.pooling = False
+
+
 def odbc_value(value: object) -> str:
     """Return *value* as an ODBC attribute value, brace-quoted when it needs it.
 

{execsql2-2.31.1 → execsql2-2.32.0}/src/execsql/db/sqlserver.py

@@ -8,7 +8,7 @@ Server via ``pyodbc``.  Corresponds to ``-t s`` on the CLI.
 """
 
 from execsql.db.base import Database
-from execsql.db.odbc import connection_string, loggable
+from execsql.db.odbc import connection_string, fresh_sessions, loggable
 from execsql.exceptions import ErrInfo
 from execsql.utils.errors import fatal_error, install_hint
 from execsql.utils.auth import (
@@ -98,6 +98,8 @@ class SqlServerDatabase(Database):
         """Open a connection to the SQL Server database via pyodbc."""
         import pyodbc
 
+        fresh_sessions(pyodbc)
+
         if self.conn is None:
             begin_login()
             if self.user and self.need_passwd and not self.password:

{execsql2-2.31.1 → execsql2-2.32.0}/src/execsql/exporters/duckdb.py

@@ -10,7 +10,6 @@ Requires the ``execsql2[duckdb]`` extra.
 
 import datetime as _datetime
 import decimal as _decimal
-import math
 from pathlib import Path
 from typing import Any
 
@@ -38,9 +37,9 @@ def export_duckdb(
         fatal_error(f"The duckdb module is required to export data in that format. {install_hint('duckdb')}")
         return
 
+    from execsql.db.duckdb import insert_rows
     from execsql.models import DataTable
 
-    chunksize = 10000
     # B07a/F026: tablename and catalog are substituted from user input.
     # Identifier-quote the table-name interpolation site and use
     # placeholders for the information_schema literal-value query.
@@ -70,15 +69,8 @@ def export_duckdb(
     curs.execute(sql)
     # Export all rows of data
     columns = [dbt_duckdb.quoted(col) for col in hdrs]
-    colspec = ",".join(columns)
-    paramspec = ",".join(("?",) * len(columns))
-    sql = f"insert into {qtable} ({colspec}) values ({paramspec});"
-    n_chunks = math.ceil(len(rowdata) / chunksize)
     curs.execute("BEGIN TRANSACTION;")
-    for i in range(n_chunks):
-        start = i * chunksize
-        end = start + chunksize
-        curs.executemany(sql, rowdata[start:end])
+    insert_rows(curs, qtable, columns, rowdata)
     curs.execute("COMMIT;")
     curs.close()
     ddb.close()

{execsql2-2.31.1 → execsql2-2.32.0}/src/execsql/models.py

@@ -50,6 +50,16 @@ __all__ = [
 ColumnType = tuple[str, type[DataType], int | None, bool, int | None, int | None]
 
 
+def _text_length(value: Any) -> int:
+    """Return the length of *value* written as text (a column may turn out to be text)."""
+    if isinstance(value, str):
+        return len(value)
+    try:
+        return len(str(value))
+    except Exception:
+        return len(bytes(value))
+
+
 class Column:
     """Compile data-type match statistics for a single column of imported data."""
 
@@ -84,40 +94,39 @@ class Column:
             """Test whether a non-null value matches this data type and update statistics."""
             # datavalue must be non-null
             if not self.failed:
-                is_match = self.dt.matches(datavalue)
-                if is_match:
-                    self.count += 1
-                    if isinstance(datavalue, str):
-                        vlen = len(datavalue)
-                    else:
-                        # This column may turn out to have to be text, so we need the maximum length
-                        # of any data value when represented as text.
-                        try:
-                            vlen = len(str(datavalue))
-                        except Exception:
-                            vlen = len(bytes(datavalue))
-                    if self.maxlen > 0 and vlen != self.maxlen:
-                        self.varlen = True
-                    if vlen > self.maxlen:
-                        self.maxlen = vlen
-                    if self.dt.precision is not None and self.dt.scale is not None:
-                        int_digits = self.dt.precision - self.dt.scale
-                        if self.maxprecision is None:
-                            self.maxprecision = self.dt.precision
-                            self.scale = self.dt.scale
-                            self.maxscale = self.dt.scale
-                            self.max_int_digits = int_digits
-                        else:
-                            assert self.maxscale is not None
-                            assert self.max_int_digits is not None
-                            self.maxprecision = max(self.dt.precision, self.maxprecision)
-                            if self.dt.scale != self.scale:
-                                self.varscale = True
-                            self.maxscale = max(self.dt.scale, self.maxscale)
-                            self.max_int_digits = max(int_digits, self.max_int_digits)
+                if self.dt.matches(datavalue):
+                    self.record(_text_length(datavalue))
                 else:
                     self.failed = True
 
+        def record(self, vlen: int) -> None:
+            """Count a value this data type matched, *vlen* characters long as text.
+
+            Call it right after :meth:`DataType.matches` accepted the value:
+            a decimal type's precision and scale are those of the value it
+            last matched.
+            """
+            self.count += 1
+            if self.maxlen > 0 and vlen != self.maxlen:
+                self.varlen = True
+            if vlen > self.maxlen:
+                self.maxlen = vlen
+            if self.dt.precision is not None and self.dt.scale is not None:
+                int_digits = self.dt.precision - self.dt.scale
+                if self.maxprecision is None:
+                    self.maxprecision = self.dt.precision
+                    self.scale = self.dt.scale
+                    self.maxscale = self.dt.scale
+                    self.max_int_digits = int_digits
+                else:
+                    assert self.maxscale is not None
+                    assert self.max_int_digits is not None
+                    self.maxprecision = max(self.dt.precision, self.maxprecision)
+                    if self.dt.scale != self.scale:
+                        self.varscale = True
+                    self.maxscale = max(self.dt.scale, self.maxscale)
+                    self.max_int_digits = max(int_digits, self.max_int_digits)
+
     def __init__(self, colname: str, infer_strings: bool = True) -> None:
         """Create a column characteriser for the named column.
 
@@ -177,6 +186,15 @@ class Column:
         self.str_accums = tuple(ac for ac in self.accums if isinstance(ac.dt, (DT_Character, DT_Varchar, DT_Text)))
         # A bytes value is checked as binary or text only: int(b"12") is 12.
         self.bytes_accums = tuple(ac for ac in self.accums if isinstance(ac.dt, DT_Binary)) + self.str_accums
+        # A value is tried only against the types that matched every value
+        # before it: a type that failed once cannot be the column's type.
+        self._live = list(self.accums)
+        self._live_str = list(self.str_accums)
+        self._live_bytes = list(self.bytes_accums)
+        # The settings the scan reads for every value, read once.
+        self._trim_strings = conf.trim_strings
+        self._replace_newlines = conf.replace_newlines
+        self._empty_strings = conf.empty_strings
         self.dt_eval = False
         # self.dt is a tuple of: 0: column name; 1: data type class; 2: max length or None;
         # 3: bool indicating any null values; 4: precision or None; 5: scale or None.
@@ -187,31 +205,38 @@ class Column:
 
     def eval_types(self, column_value: Any) -> None:
         """Evaluate which data types the value matches and update counters."""
-        # Evaluate which data type(s) the value matches, and increment the appropriate counter(s).
-        import execsql.state as _state
-
         self.dt_eval = False
         self.rowcount += 1
-        conf = _state.conf
-        if column_value is not None and isinstance(column_value, str):
-            if conf.trim_strings:
+        is_str = isinstance(column_value, str)
+        if is_str:
+            if self._trim_strings:
                 column_value = column_value.strip()
-            if conf.replace_newlines:
+            if self._replace_newlines:
                 column_value = re.sub(r"[\s\t]*[\r\n]+[\s\t]*", " ", column_value)
-        if column_value is None or (
-            not conf.empty_strings and isinstance(column_value, str) and len(column_value.strip()) == 0
-        ):
+        if column_value is None or (not self._empty_strings and is_str and len(column_value.strip()) == 0):
             self.nullrows += 1
             return
-        if not self.infer_strings and isinstance(column_value, str):
-            for dt in self.str_accums:
-                dt.check(column_value)
+        if not self.infer_strings and is_str:
+            live = self._live_str
         elif not self.infer_strings and isinstance(column_value, (bytes, bytearray, memoryview)):
-            for dt in self.bytes_accums:
-                dt.check(column_value)
+            live = self._live_bytes
         else:
-            for dt in self.accums:
-                dt.check(column_value)
+            live = self._live
+        vlen = None
+        any_failed = False
+        # The value is not NULL, the one case DataType.matches() adds to _is_match().
+        for ac in live:
+            if ac.dt._is_match(column_value):
+                if vlen is None:
+                    vlen = _text_length(column_value)
+                ac.record(vlen)
+            else:
+                ac.failed = True
+                any_failed = True
+        if any_failed:
+            self._live = [ac for ac in self._live if not ac.failed]
+            self._live_str = [ac for ac in self._live_str if not ac.failed]
+            self._live_bytes = [ac for ac in self._live_bytes if not ac.failed]
 
     def column_type(self) -> ColumnType:
         """Return the inferred type of this column as a 6-tuple."""

{execsql2-2.31.1 → execsql2-2.32.0}/src/execsql/types.py

@@ -550,6 +550,8 @@ class DT_Character(DataType):
         return "DT_Character()"
 
     def _is_match(self, data: object) -> bool:
+        if isinstance(data, str):  # what _from_data() returns a str as, unchanged
+            return True
         if isinstance(data, bytearray):
             return False
         return super()._is_match(data)
@@ -582,6 +584,8 @@ class DT_Varchar(DataType):
         return "DT_Varchar()"
 
     def _is_match(self, data: object) -> bool:
+        if isinstance(data, str):  # the length check _from_data() makes
+            return len(data) <= 255
         if isinstance(data, bytearray):
             return False
         return super()._is_match(data)
@@ -609,6 +613,8 @@ class DT_Text(DataType):
         return "DT_Text()"
 
     def _is_match(self, data: object) -> bool:
+        if isinstance(data, str):  # what _from_data() returns a str as, unchanged
+            return True
         if isinstance(data, bytearray):
             return False
         return super()._is_match(data)
@@ -634,6 +640,14 @@ class DT_Binary(DataType):
     def __repr__(self) -> str:
         return "DT_Binary()"
 
+    def _is_match(self, data: object) -> bool:
+        # bytearray(n) is n zero bytes, so an int converts when it is not
+        # negative.  Building it to find out allocated n bytes per value: a
+        # column of epoch milliseconds asked for a terabyte each.
+        if isinstance(data, int):
+            return data >= 0
+        return super()._is_match(data)
+
 
 class DbType:
     """Map Python DataType subclasses to DBMS-specific SQL type names."""