execsql2 2.15.5__tar.gz → 2.15.7__tar.gz

{execsql2-2.15.5 → execsql2-2.15.7}/CHANGELOG.md

@@ -13,6 +13,38 @@ ______________________________________________________________________
 
 ______________________________________________________________________
 
+## [2.15.7] - 2026-04-20
+
+### Fixed
+
+- `CounterVars.substitute` now correctly searches the full string. `re.I` was mistakenly passed as the `pos` argument to `re.search`, causing the first two characters of every string to be skipped when looking for counter variable references.
+- `DataTypeError`, `DbTypeError`, and `DatabaseNotImplementedError` now call `super().__init__()` instead of bypassing the MRO with `Exception.__init__(self, ...)`, fixing `repr()` crashes for these exception types.
+- SQL injection in MySQL `LOAD DATA INFILE`: file path, field delimiter, and quote character are now escaped before being interpolated into the import SQL statement.
+- SQLite and DuckDB `exec_cmd` no longer encodes the SQL string to bytes before passing it to `execute()`, which always raised `TypeError` in Python 3.
+- Substitution variable token matching in `_substitute_nested` now uses `str.find()` on a lower-cased copy of the string instead of compiling a new regex per variable per call, eliminating unnecessary regex compilation on every substitution.
+- Cursor leaks across all database adapters (PostgreSQL, MySQL, SQLite, DuckDB, Firebird, Access, SQL Server, Oracle) — call sites that manually opened cursors now use the `with self._cursor()` context manager so cursors are always closed, including on exceptions.
+- `__delattr__` in `state.py` now instantiates a fresh `RuntimeContext()` when resetting an attribute to its default, rather than reading from a cached `_DEFAULT_CTX` instance. This prevents mutable defaults (lists, dicts) from being shared across resets.
+- `cmds_run` counter no longer overcounts: the `StopIteration` branch in `runscripts()` now calls `continue` after popping the command list stack, preventing the increment that follows from executing.
+- Config file chaining is now capped at 20 files to prevent an infinite loop when `config_file` entries form a cycle.
+- Temp file creation in `TempFileMgr` now uses `tempfile.mkstemp()` instead of `tempfile.NamedTemporaryFile().name`, eliminating the TOCTOU race where another process could claim the name between creation and use.
+- JSON export now serializes column names with `json.dumps()` instead of bare f-string interpolation, preventing malformed JSON when column names contain quotes, backslashes, or other special characters.
+- PostgreSQL `VACUUM` autocommit session state is now restored in a `finally` block, ensuring the connection returns to non-autocommit mode even if the vacuum statement raises an exception.
+- HTML export now HTML-escapes the description, author, and CSS href meta tag values, preventing malformed HTML when these values contain `<`, `>`, `"`, or `&` characters.
+- `shlex.split` on Windows is now called with `posix=False` instead of pre-escaping backslashes, which produced incorrect splits for paths with consecutive backslashes.
+- Duplicate `JsonDatatype.integer = "integer"` assignment in `models.py` removed; `JsonDatatype.number` is the correct attribute and was already present.
+- MySQL adapter constructor no longer coerces `None` arguments to the string `"None"` for `server_name`, `db_name`, and `user_name`; `None` values are now preserved as `None`.
+- `importfile()` parameter renamed from `columname` to `column_name`, matching the internal variable name used throughout the function body.
+
+______________________________________________________________________
+
+## [2.15.6] - 2026-04-16
+
+### Fixed
+
+- Nested substitution variable names (e.g., `!!N_!!CHECK_GROUP!!_CHECKS!!`) now resolve correctly, matching original execsql behavior. The single-pass token regex introduced in 2.15.0 could not find inner `!!var!!` tokens embedded within an outer variable name; a per-variable substring fallback now handles this edge case.
+
+______________________________________________________________________
+
 ## [2.15.5] - 2026-04-15
 
 ### Fixed

{execsql2-2.15.5 → execsql2-2.15.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: execsql2
-Version: 2.15.5
+Version: 2.15.7
 Summary: Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables.
 Project-URL: Homepage, https://execsql2.readthedocs.io
 Project-URL: Repository, https://github.com/geocoug/execsql

{execsql2-2.15.5 → execsql2-2.15.7}/docs/about/divergence.md

@@ -210,6 +210,7 @@ These are behavioral changes driven by security or correctness issues in the ups
 | Database metadata queries    | `schema_exists()`, `table_exists()`, `column_exists()`, `table_columns()`, `view_exists()`, `role_exists()` across all 9 adapters now use parameterized queries. Upstream used string interpolation. |
 | `import_entire_file()`       | Column names are quoted with `quote_identifier()` instead of interpolated into INSERT statements.                                                                                                    |
 | PostgreSQL `CREATE DATABASE` | Database name and encoding are quoted. COPY delimiter and quote character are validated.                                                                                                             |
+| MySQL `LOAD DATA INFILE`     | File path, delimiter, and quotechar are now escaped with `replace("'", "''")` before interpolation into the SQL statement.                                                                           |
 | `$SHEETS_TABLES_VALUES`      | Sheet names from ODS/XLS imports are escaped before embedding in SQL.                                                                                                                                |
 | HTTP `Content-Disposition`   | Filename is sanitized to prevent HTTP response splitting in SERVE.                                                                                                                                   |
 
@@ -220,7 +221,7 @@ These are behavioral changes driven by security or correctness issues in the ups
 | Jinja2 sandboxing | Templates run in `SandboxedEnvironment` instead of the default `jinja2.Template`.                      |
 | HTML export       | Column headers and cell values are escaped with `html.escape()` to prevent XSS.                        |
 | XML export        | Values are escaped with `xml.sax.saxutils.escape()`. Invalid XML element name characters are replaced. |
-| JSON export       | The `description` field uses `json.dumps()` instead of string interpolation.                           |
+| JSON export       | The `description` field and all column names use `json.dumps()` instead of string interpolation.       |
 
 ### Credential and Logging Safety
 
@@ -254,6 +255,14 @@ These are behavioral changes driven by security or correctness issues in the ups
 | `NumericParser` division by zero                    | `NumericAstNode.eval()` raised unhandled `ZeroDivisionError`. Now raises `NumericParserError` with a clear message. Inherited from upstream.                                                                                                                                                           |
 | `CondAstNode.eval()` could return `None`            | Missing fallthrough for unknown node types silently returned `None`. Now raises `CondParserError`. Inherited from upstream.                                                                                                                                                                            |
 | `DT_Timestamp` claims time-only values              | `dateutil.parser.parse()` silently fills in today's date for bare time strings like `"13:15:45"`, so `DT_Timestamp` matched before `DT_Time` in the inference order. `parse_datetime()` now rejects time-only strings. Inherited from upstream.                                                        |
+| `CounterVars.substitute` skipped pos 0–1            | `re.I` was passed as the positional `pos` argument, skipping the first 2 characters of every string during counter variable expansion. Counter variables at the very start of a line were never matched. Inherited from upstream.                                                                      |
+| `exec_cmd` (SQLite/DuckDB) always raised TypeError  | Passed bytes to `curs.execute()` via `.encode()`, which Python 3 `sqlite3`/`duckdb` reject. `EXECUTE PROCEDURE` metacommand was non-functional on these backends. Inherited from upstream.                                                                                                             |
+| MySQL `LOAD DATA INFILE` injection                  | File path, delimiter, and quotechar were interpolated without escaping. Now single-quotes are escaped consistent with the PostgreSQL COPY path. Inherited from upstream.                                                                                                                               |
+| Config file chain infinite loop                     | The `config_file` directive could chain config files without limit. A circular reference (via symlinks or different relative paths) caused an infinite loop at startup. Now capped at 20 files. Inherited from upstream.                                                                               |
+| Cursor leaks in database adapters                   | ~15 methods across all adapters used `curs = self.cursor()` / `curs.close()` without `try/finally`. If the query raised, the cursor leaked. Converted to `with self._cursor() as curs:`. Inherited from upstream.                                                                                      |
+| JSON export malformed on special column names       | Column names containing `"` or `\` produced invalid JSON. Now uses `json.dumps()` for all field names. Inherited from upstream.                                                                                                                                                                        |
+| Temp file creation TOCTOU race                      | `TempFileMgr.new_temp_fn()` discarded the `NamedTemporaryFile` handle, creating a race window. Now uses `tempfile.mkstemp()` for secure creation. Inherited from upstream.                                                                                                                             |
+| `shlex.split` on Windows incorrect mode             | Called without `posix=False` on Windows, mishandling backslash-heavy paths in SHELL commands. Inherited from upstream.                                                                                                                                                                                 |
 
 ______________________________________________________________________
 

{execsql2-2.15.5 → execsql2-2.15.7}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "execsql2"
-version = "2.15.5"
+version = "2.15.7"
 description = "Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables."
 readme = { file = "README.md", content-type = "text/markdown" }
 license = { file = "LICENSE.txt" }
@@ -164,7 +164,7 @@ skip-magic-trailing-comma = false
 line-ending = "auto"
 
 [tool.bumpversion]
-current_version = "2.15.5"
+current_version = "2.15.7"
 commit = true
 commit_args = "--no-verify"
 tag = true

{execsql2-2.15.5 → execsql2-2.15.7}/src/execsql/config.py

@@ -290,8 +290,11 @@ class ConfigData:
             config_files = [sys_config_file, user_config_file, script_config_file, startdir_config_file]
         else:
             config_files = [sys_config_file, user_config_file, startdir_config_file]
+        _MAX_CONFIG_CHAIN = 20  # Guard against circular config_file references.
         self.files_read: list = []
         for ix, configfile in enumerate(config_files):
+            if len(self.files_read) >= _MAX_CONFIG_CHAIN:
+                break
             if configfile not in self.files_read and Path(configfile).is_file():
                 self.files_read.append(configfile)
                 cp = ConfigParser()

{execsql2-2.15.5 → execsql2-2.15.7}/src/execsql/db/access.py

@@ -243,16 +243,16 @@ class AccessDatabase(Database):
                     self.temp_query_names.append(qn)
             else:
                 self.dao_flush_check()
-                curs = self.cursor()
-                if self.jet4:
-                    encoded_sql = str(sql)
-                else:
-                    encoded_sql = str(sql).encode(self.encoding)
-                if paramlist is None:
-                    curs.execute(encoded_sql)
-                else:
-                    curs.execute(encoded_sql, paramlist)
-                _state.subvars.add_substitution("$LAST_ROWCOUNT", curs.rowcount)
+                with self._cursor() as curs:
+                    if self.jet4:
+                        encoded_sql = str(sql)
+                    else:
+                        encoded_sql = str(sql).encode(self.encoding)
+                    if paramlist is None:
+                        curs.execute(encoded_sql)
+                    else:
+                        curs.execute(encoded_sql, paramlist)
+                    _state.subvars.add_substitution("$LAST_ROWCOUNT", curs.rowcount)
 
         if type(sqlcmd) in (list, tuple):
             for sql in sqlcmd:
@@ -269,10 +269,10 @@ class AccessDatabase(Database):
         # Returns the results of the sql select statement.
         # The Access driver returns data as unicode, so no decoding is necessary.
         self.dao_flush_check()
-        curs = self.cursor()
-        curs.execute(sql)
-        rows = curs.fetchall()
-        return [d[0] for d in curs.description], rows
+        with self._cursor() as curs:
+            curs.execute(sql)
+            rows = curs.fetchall()
+            return [d[0] for d in curs.description], rows
 
     def select_rowsource(self, sql: str) -> tuple[list[str], Any]:
         """Return column names and an iterable that yields rows one at a time."""
@@ -308,20 +308,20 @@ class AccessDatabase(Database):
     def table_exists(self, table_name: str, schema_name: str | None = None) -> bool:
         """Return True if the named table exists in the Access database."""
         self.dao_flush_check()
-        curs = self.cursor()
-        try:
-            sql = "select Name from MSysObjects where Name=? And Type In (1,4,6);"
-            curs.execute(sql, (table_name,))
-        except ErrInfo:
-            raise
-        except Exception as e:
-            raise ErrInfo(
-                type="db",
-                command_text=sql,
-                exception_msg=exception_desc(),
-                other_msg=f"Failure on test for existence of Access table {table_name}",
-            ) from e
-        rows = curs.fetchall()
+        sql = "select Name from MSysObjects where Name=? And Type In (1,4,6);"
+        with self._cursor() as curs:
+            try:
+                curs.execute(sql, (table_name,))
+            except ErrInfo:
+                raise
+            except Exception as e:
+                raise ErrInfo(
+                    type="db",
+                    command_text=sql,
+                    exception_msg=exception_desc(),
+                    other_msg=f"Failure on test for existence of Access table {table_name}",
+                ) from e
+            rows = curs.fetchall()
         return len(rows) > 0
 
     def column_exists(
@@ -332,41 +332,41 @@ class AccessDatabase(Database):
     ) -> bool:
         """Return True if the named column exists in the given Access table."""
         self.dao_flush_check()
-        curs = self.cursor()
         quoted_col = self.quote_identifier(column_name)
         quoted_tbl = self.quote_identifier(table_name)
         sql = f"select top 1 {quoted_col} from {quoted_tbl};"
-        try:
-            curs.execute(sql)
-        except Exception:
-            return False
+        with self._cursor() as curs:
+            try:
+                curs.execute(sql)
+            except Exception:
+                return False
         return True
 
     def table_columns(self, table_name: str, schema_name: str | None = None) -> list[str]:
         """Return a list of column names for the given Access table."""
         self.dao_flush_check()
-        curs = self.cursor()
         quoted_tbl = self.quote_identifier(table_name)
-        curs.execute(f"select top 1 * from {quoted_tbl};")
-        return [d[0] for d in curs.description]
+        with self._cursor() as curs:
+            curs.execute(f"select top 1 * from {quoted_tbl};")
+            return [d[0] for d in curs.description]
 
     def view_exists(self, view_name: str, schema_name: str | None = None) -> bool:
         """Return True if the named view or query exists in the Access database."""
         self.dao_flush_check()
-        curs = self.cursor()
-        try:
-            sql = "select Name from MSysObjects where Name=? And Type = 5;"
-            curs.execute(sql, (view_name,))
-        except ErrInfo:
-            raise
-        except Exception as e:
-            raise ErrInfo(
-                type="db",
-                command_text=sql,
-                exception_msg=exception_desc(),
-                other_msg=f"Test for existence of Access view/query {view_name}",
-            ) from e
-        rows = curs.fetchall()
+        sql = "select Name from MSysObjects where Name=? And Type = 5;"
+        with self._cursor() as curs:
+            try:
+                curs.execute(sql, (view_name,))
+            except ErrInfo:
+                raise
+            except Exception as e:
+                raise ErrInfo(
+                    type="db",
+                    command_text=sql,
+                    exception_msg=exception_desc(),
+                    other_msg=f"Test for existence of Access view/query {view_name}",
+                ) from e
+            rows = curs.fetchall()
         return len(rows) > 0
 
     def schema_exists(self, schema_name: str) -> bool:
@@ -447,4 +447,5 @@ class AccessDatabase(Database):
         sq_name = self.schema_qualified_table_name(schema_name, table_name)
         quoted_col = self.quote_identifier(column_name)
         sql = f"insert into {sq_name} ({quoted_col}) values ({self.paramsubs(1)});"
-        self.cursor().execute(sql, (pyodbc.Binary(filedata),))
+        with self._cursor() as curs:
+            curs.execute(sql, (pyodbc.Binary(filedata),))

{execsql2-2.15.5 → execsql2-2.15.7}/src/execsql/db/base.py

@@ -460,7 +460,6 @@ class Database(ABC):
         paramspec = self.paramsubs(len(columns))
         sql = f"insert into {sq_name} ({colspec}) values ({paramspec});"
         rows = iter(rowsource)
-        curs = self.cursor()
         eof = False
         total_rows = 0
 
@@ -483,7 +482,7 @@ class Database(ABC):
             except ImportError:
                 use_progress = False
 
-        def _import_loop() -> int:
+        def _import_loop(curs) -> int:
             nonlocal eof, total_rows, task_id
             while True:
                 b = []
@@ -575,12 +574,13 @@ class Database(ABC):
                     break
             return total_rows
 
-        if use_progress and progress_ctx is not None:
-            with progress_ctx:
-                task_id = progress_ctx.add_task(sq_name, total=None)
-                _import_loop()
-        else:
-            _import_loop()
+        with self._cursor() as curs:
+            if use_progress and progress_ctx is not None:
+                with progress_ctx:
+                    task_id = progress_ctx.add_task(sq_name, total=None)
+                    _import_loop(curs)
+            else:
+                _import_loop(curs)
 
         if _state.exec_log:
             _state.exec_log.log_status_info(

{execsql2-2.15.5 → execsql2-2.15.7}/src/execsql/db/dsn.py

@@ -145,4 +145,5 @@ class DsnDatabase(Database):
         sq_name = self.schema_qualified_table_name(schema_name, table_name)
         quoted_col = self.quote_identifier(column_name)
         sql = f"insert into {sq_name} ({quoted_col}) values ({self.paramsubs(1)});"
-        self.cursor().execute(sql, (pyodbc.Binary(filedata),))
+        with self._cursor() as curs:
+            curs.execute(sql, (pyodbc.Binary(filedata),))

{execsql2-2.15.5 → execsql2-2.15.7}/src/execsql/db/duckdb.py

@@ -67,7 +67,7 @@ class DuckDBDatabase(Database):
         with self._cursor() as curs:
             cmd = f"select * from {querycommand};"
             try:
-                curs.execute(cmd.encode(self.encoding))
+                curs.execute(cmd)
                 _state.subvars.add_substitution("$LAST_ROWCOUNT", curs.rowcount)
             except Exception:
                 self.rollback()

{execsql2-2.15.5 → execsql2-2.15.7}/src/execsql/db/firebird.py

@@ -127,30 +127,29 @@ class FirebirdDatabase(Database):
 
     def table_exists(self, table_name: str, schema_name: str | None = None) -> bool:
         """Return True if the named table exists in the Firebird database."""
-        curs = self.cursor()
         sql = (
             "SELECT RDB$RELATION_NAME FROM RDB$RELATIONS "
             "WHERE RDB$SYSTEM_FLAG=0 AND RDB$VIEW_BLR IS NULL "
             "AND RDB$RELATION_NAME=?;"
         )
-        try:
-            curs.execute(sql, (table_name.upper(),))
-        except ErrInfo:
-            raise
-        except Exception as e:
+        with self._cursor() as curs:
             try:
-                self.rollback()
-            except Exception:
-                pass  # Rollback is best-effort after a failed query.
-            raise ErrInfo(
-                type="db",
-                command_text=sql,
-                exception_msg=exception_desc(),
-                other_msg=f"Failed test for existence of Firebird table {table_name}",
-            ) from e
-        rows = curs.fetchall()
+                curs.execute(sql, (table_name.upper(),))
+            except ErrInfo:
+                raise
+            except Exception as e:
+                try:
+                    self.rollback()
+                except Exception:
+                    pass  # Rollback is best-effort after a failed query.
+                raise ErrInfo(
+                    type="db",
+                    command_text=sql,
+                    exception_msg=exception_desc(),
+                    other_msg=f"Failed test for existence of Firebird table {table_name}",
+                ) from e
+            rows = curs.fetchall()
         self.conn.commit()
-        curs.close()
         return len(rows) > 0
 
     def column_exists(
@@ -160,53 +159,52 @@ class FirebirdDatabase(Database):
         schema_name: str | None = None,
     ) -> bool:
         """Return True if the named column exists in the given Firebird table."""
-        curs = self.cursor()
         quoted_col = self.quote_identifier(column_name)
         quoted_tbl = self.quote_identifier(table_name)
         sql = f"select first 1 {quoted_col} from {quoted_tbl};"
-        try:
-            curs.execute(sql)
-        except Exception:
-            return False
+        with self._cursor() as curs:
+            try:
+                curs.execute(sql)
+            except Exception:
+                return False
         return True
 
     def table_columns(self, table_name: str, schema_name: str | None = None) -> list[str]:
         """Return a list of column names for the given Firebird table."""
-        curs = self.cursor()
         quoted_tbl = self.quote_identifier(table_name)
         sql = f"select first 1 * from {quoted_tbl};"
-        try:
-            curs.execute(sql)
-        except ErrInfo:
-            raise
-        except Exception as e:
-            self.rollback()
-            raise ErrInfo(
-                type="db",
-                command_text=sql,
-                exception_msg=exception_desc(),
-                other_msg=f"Failed to get column names for table {table_name} of {self.name()}",
-            ) from e
-        return [d[0] for d in curs.description]
+        with self._cursor() as curs:
+            try:
+                curs.execute(sql)
+            except ErrInfo:
+                raise
+            except Exception as e:
+                self.rollback()
+                raise ErrInfo(
+                    type="db",
+                    command_text=sql,
+                    exception_msg=exception_desc(),
+                    other_msg=f"Failed to get column names for table {table_name} of {self.name()}",
+                ) from e
+            return [d[0] for d in curs.description]
 
     def view_exists(self, view_name: str, schema_name: str | None = None) -> bool:
         """Return True if the named view exists in the Firebird database."""
-        curs = self.cursor()
         sql = "select distinct rdb$view_name from rdb$view_relations where rdb$view_name = ?;"
-        try:
-            curs.execute(sql, (view_name,))
-        except ErrInfo:
-            raise
-        except Exception as e:
-            self.rollback()
-            raise ErrInfo(
-                type="db",
-                command_text=sql,
-                exception_msg=exception_desc(),
-                other_msg=f"Failed test for existence of Firebird view {view_name}",
-            ) from e
-        rows = curs.fetchall()
-        curs.close()
+        with self._cursor() as curs:
+            try:
+                curs.execute(sql, (view_name,))
+            except ErrInfo:
+                raise
+            except Exception as e:
+                self.rollback()
+                raise ErrInfo(
+                    type="db",
+                    command_text=sql,
+                    exception_msg=exception_desc(),
+                    other_msg=f"Failed test for existence of Firebird view {view_name}",
+                ) from e
+            rows = curs.fetchall()
         return len(rows) > 0
 
     def schema_exists(self, schema_name: str) -> bool:
@@ -215,14 +213,13 @@ class FirebirdDatabase(Database):
 
     def role_exists(self, rolename: str) -> bool:
         """Return True if the named role or user exists in the Firebird database."""
-        curs = self.cursor()
-        curs.execute(
-            "SELECT DISTINCT USER FROM RDB$USER_PRIVILEGES WHERE USER = ? union "
-            " SELECT DISTINCT RDB$ROLE_NAME FROM RDB$ROLES WHERE RDB$ROLE_NAME = ?;",
-            (rolename, rolename),
-        )
-        rows = curs.fetchall()
-        curs.close()
+        with self._cursor() as curs:
+            curs.execute(
+                "SELECT DISTINCT USER FROM RDB$USER_PRIVILEGES WHERE USER = ? union "
+                " SELECT DISTINCT RDB$ROLE_NAME FROM RDB$ROLES WHERE RDB$ROLE_NAME = ?;",
+                (rolename, rolename),
+            )
+            rows = curs.fetchall()
         return len(rows) > 0
 
     def drop_table(self, tablename: str) -> None: