PyPI - execsql2 - Versions diffs - 2.16.0__tar.gz → 2.16.1__tar.gz - Mend

execsql2 2.16.0tar.gz → 2.16.1tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (321) hide show

{execsql2-2.16.0 → execsql2-2.16.1}/.gitignore RENAMED Viewed

@@ -46,6 +46,7 @@ _execsql/
 # Claude
 .claude/
-ANALYSIS.md
-FINDINGS.md
 CLAUDE.md
+AUDIT.md
+DOC_AUDIT.md
+PROMPTS.md

{execsql2-2.16.0 → execsql2-2.16.1}/CHANGELOG.md RENAMED Viewed

@@ -13,6 +13,20 @@ ______________________________________________________________________
 ______________________________________________________________________
+## [2.16.1] - 2026-04-30
+### Fixed
+- AST executor now correctly handles `~` (local) and `+` (outer-scope) substitution variables inside SCRIPT blocks. Previously, `-- !x! SUB ~var value` inside a SCRIPT body would write to a disconnected scope, causing the variable to be invisible to subsequent SQL statements and producing spurious "potential un-substituted variable" warnings. The fix pushes proper `CommandList` frames onto `commandliststack` at script and top-level boundaries, bridging the AST executor with the legacy metacommand handlers (`x_sub`, `x_rm_sub`, `xf_sub_defined`, `SUB_LOCAL`, prompt handlers, REPL `.vars`/`.stack`, etc.).
+- EXECUTE SCRIPT argument expressions (e.g., `val=!!#parent_param!!`) are now expanded in the caller's scope before the child script frame is created, fixing nested script calls that pass `~` or `#` variables as arguments.
+### Changed
+- `RuntimeContext` is now stored in `threading.local()` instead of a module-level global, making `_state.foo` access thread-safe. Each thread gets its own isolated context via lazy initialization. The `active_context` context manager is now safe for concurrent use across threads. Enables future PARALLEL blocks and concurrent `from execsql import run` calls.
+- `_run()` in `cli/run.py` decomposed into 8 standalone functions: `_seed_early_subvars()`, `_load_config()`, `_seed_script_subvars()`, `_load_script()`, `_apply_dsn()`, `_apply_cli_options()`, `_route_positionals()`, and `_setup_logging()`. Reduces `_run()` from ~380 lines to ~150 lines of orchestration with zero behavioral change.
+______________________________________________________________________
 ## [2.16.0] - 2026-04-29
 ### Added
@@ -37,6 +51,7 @@ ______________________________________________________________________
 - `--lint` now uses the AST parser for structural validation. Unmatched IF/LOOP/BATCH/SCRIPT blocks are caught at parse time with precise source line ranges. No database connection or runtime state initialization is required. All prior lint checks (variable analysis, INCLUDE file existence, EXECUTE SCRIPT resolution, SUB_INI reading) are preserved.
 - Export format dispatch logic (`EXPORT` and `EXPORT QUERY` metacommands) refactored from duplicated ~180-line if/elif chains into shared `_dispatch_format()` function, eliminating code duplication and fixing missing zip-compatibility checks for `EXPORT QUERY`.
 - `MailSpec.send()` refactored: extracted `_expand()` helper to replace 12 repetitive substitution lines.
+- Default database type changed from Access (`-t a`) to SQLite (`-t l`). Upstream defaulted to Access, which requires Windows and pyodbc. Users targeting Access databases should pass `-t a` explicitly.
 ### Fixed
@@ -55,6 +70,19 @@ ______________________________________________________________________
 - `JsonDatatype` attributes are now declared as class variables in the class body instead of assigned externally after class definition.
 - `minimal_conf` test fixture expanded with commonly needed attributes (`import_encoding`, `script_encoding`, `export_output_dir`, `write_prefix`, `write_suffix`, `fold_col_hdrs`, `trim_col_hdrs`, etc.) to reduce ad-hoc attribute additions in individual tests.
+### Documentation
+- Fixed false `$ENV:` prefix claim in substitution variables reference — feature does not exist.
+- Documented environment variable filtering (SECRET, TOKEN, PASSWORD, etc.) in substitution variables reference.
+- Added missing exporter API docs (markdown, yaml, xlsx) and importer API docs (json).
+- Added 8 missing CLI flags to README Options table (-b, -e, -g, -i, -o, -s, -y, -z).
+- Added missing installation extras ([upsert], [firebird], [oracle]) to README and installation guide.
+- Fixed broken `PROMPT.md` link in logging guide.
+- Added explicit `{ #exampleN }` anchors to all 34 examples for reliable cross-referencing.
+- Updated architecture doc: corrected metacommand count (~225), export format count (20+), added debug/notebook/server/lsp packages to module map.
+- Updated metacommand developer guide to reflect io.py split into io_export.py, io_import.py, io_write.py, io_fileops.py.
+- Noted SQLite as the default database type in syntax reference.
 ### Removed
 - `--ast` / `--no-ast` CLI flag — the AST executor is now the only execution engine; no opt-out.

{execsql2-2.16.0 → execsql2-2.16.1}/PKG-INFO RENAMED Viewed

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: execsql2
-Version: 2.16.0
+Version: 2.16.1
 Summary: Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables.
 Project-URL: Homepage, https://execsql2.readthedocs.io
 Project-URL: Repository, https://github.com/geocoug/execsql
@@ -173,7 +173,8 @@ pip install execsql2[oracle]      # Oracle (oracledb)
 pip install execsql2[odbc]        # ODBC DSN (pyodbc)
 # Feature bundles
-pip install execsql2[formats]    # ODS, Excel, Jinja2, Feather, Parquet, HDF5
+pip install execsql2[formats]         # ODS, Excel, Jinja2, Feather, Parquet, HDF5
+pip install execsql2[upsert]          # pg-upsert for PostgreSQL upsert operations
 pip install execsql2[auth]            # OS keyring integration
 pip install execsql2[auth-plaintext]  # Keyring + plaintext file backend (headless Linux)
 pip install execsql2[auth-encrypted]  # Keyring + encrypted file backend (headless Linux)
@@ -219,33 +220,41 @@ execsql script.sql                          # read connection from config file
 ## Options
-| Flag                                | Description                                                     |
-| ----------------------------------- | --------------------------------------------------------------- |
-| `-t {p,m,s,l,k,a,f,o,d}`            | Database type                                                   |
-| `-u USER`                           | Database username                                               |
-| `-p PORT`                           | Server port                                                     |
-| `-a VALUE`                          | Set substitution variable `$ARG_x`                              |
-| `-c SCRIPT`                         | Execute inline SQL or metacommand string                        |
-| `-d`                                | Auto-create export directories                                  |
-| `-f ENCODING`                       | Script file encoding (default: UTF-8)                           |
-| `-l`                                | Write run log to `~/execsql.log`                                |
-| `-m`                                | List metacommands and exit                                      |
-| `-n`                                | Create a new SQLite or PostgreSQL database if it does not exist |
-| `-v {0,1,2,3}`                      | GUI level (0=none, 1=password, 2=selection, 3=full)             |
-| `-w`                                | Skip password prompt when a username is supplied                |
-| `--dsn URL`                         | Connection string (e.g. `postgresql://user:pass@host/db`)       |
-| `--output-dir DIR`                  | Default base directory for EXPORT output files                  |
-| `--dry-run`                         | Parse the script and report commands without executing          |
-| `--lint`                            | Static analysis: check structure and warn on issues (no DB)     |
-| `--parse-tree`                      | Print the script's AST structure and exit (no DB)               |
-| `--list-plugins`                    | List discovered plugins and exit                                |
-| `--ping`                            | Test database connectivity and exit                             |
-| `--profile`                         | Show per-statement timing summary after execution               |
-| `--progress`                        | Show a progress bar for long-running IMPORT operations          |
-| `--config FILE`                     | Load an explicit config file (highest priority after CLI args)  |
-| `--debug`                           | Start in step-through debug mode (REPL pauses before each stmt) |
-| `--dump-keywords`                   | Print metacommand keywords as JSON and exit                     |
-| `--gui-framework {tkinter,textual}` | GUI framework for interactive prompts                           |
+| Flag                                  | Description                                                      |
+| ------------------------------------- | ---------------------------------------------------------------- |
+| `-t {p,m,s,l,k,a,f,o,d}`              | Database type                                                    |
+| `-u USER`                             | Database username                                                |
+| `-p PORT`                             | Server port                                                      |
+| `-a VALUE`                            | Set substitution variable `$ARG_x`                               |
+| `-b` / `--boolean-int`                | Treat integers 0 and 1 as boolean values                         |
+| `-c SCRIPT`                           | Execute inline SQL or metacommand string                         |
+| `-d`                                  | Auto-create export directories                                   |
+| `-e ENCODING` / `--database-encoding` | Character encoding used in the database                          |
+| `-f ENCODING`                         | Script file encoding (default: UTF-8)                            |
+| `-g ENCODING` / `--output-encoding`   | Encoding for WRITE and EXPORT output                             |
+| `-i ENCODING` / `--import-encoding`   | Encoding for data files used with IMPORT                         |
+| `-l`                                  | Write run log to `~/execsql.log`                                 |
+| `-m`                                  | List metacommands and exit                                       |
+| `-n`                                  | Create a new SQLite or PostgreSQL database if it does not exist  |
+| `-o` / `--online-help`                | Open the online documentation in the default browser             |
+| `-s N` / `--scan-lines`               | Lines to scan for IMPORT format detection (0 = scan entire file) |
+| `-v {0,1,2,3}`                        | GUI level (0=none, 1=password, 2=selection, 3=full)              |
+| `-w`                                  | Skip password prompt when a username is supplied                 |
+| `-y` / `--encodings`                  | List available encoding names and exit                           |
+| `-z KB` / `--import-buffer`           | Import buffer size in KB (default: 32)                           |
+| `--dsn URL`                           | Connection string (e.g. `postgresql://user:pass@host/db`)        |
+| `--output-dir DIR`                    | Default base directory for EXPORT output files                   |
+| `--dry-run`                           | Parse the script and report commands without executing           |
+| `--lint`                              | Static analysis: check structure and warn on issues (no DB)      |
+| `--parse-tree`                        | Print the script's AST structure and exit (no DB)                |
+| `--list-plugins`                      | List discovered plugins and exit                                 |
+| `--ping`                              | Test database connectivity and exit                              |
+| `--profile`                           | Show per-statement timing summary after execution                |
+| `--progress`                          | Show a progress bar for long-running IMPORT operations           |
+| `--config FILE`                       | Load an explicit config file (highest priority after CLI args)   |
+| `--debug`                             | Start in step-through debug mode (REPL pauses before each stmt)  |
+| `--dump-keywords`                     | Print metacommand keywords as JSON and exit                      |
+| `--gui-framework {tkinter,textual}`   | GUI framework for interactive prompts                            |
 Run `execsql --help` for the full option list, or `execsql -m` to list all metacommands.

{execsql2-2.16.0 → execsql2-2.16.1}/README.md RENAMED Viewed

@@ -51,7 +51,8 @@ pip install execsql2[oracle]      # Oracle (oracledb)
 pip install execsql2[odbc]        # ODBC DSN (pyodbc)
 # Feature bundles
-pip install execsql2[formats]    # ODS, Excel, Jinja2, Feather, Parquet, HDF5
+pip install execsql2[formats]         # ODS, Excel, Jinja2, Feather, Parquet, HDF5
+pip install execsql2[upsert]          # pg-upsert for PostgreSQL upsert operations
 pip install execsql2[auth]            # OS keyring integration
 pip install execsql2[auth-plaintext]  # Keyring + plaintext file backend (headless Linux)
 pip install execsql2[auth-encrypted]  # Keyring + encrypted file backend (headless Linux)
@@ -97,33 +98,41 @@ execsql script.sql                          # read connection from config file
 ## Options
-| Flag                                | Description                                                     |
-| ----------------------------------- | --------------------------------------------------------------- |
-| `-t {p,m,s,l,k,a,f,o,d}`            | Database type                                                   |
-| `-u USER`                           | Database username                                               |
-| `-p PORT`                           | Server port                                                     |
-| `-a VALUE`                          | Set substitution variable `$ARG_x`                              |
-| `-c SCRIPT`                         | Execute inline SQL or metacommand string                        |
-| `-d`                                | Auto-create export directories                                  |
-| `-f ENCODING`                       | Script file encoding (default: UTF-8)                           |
-| `-l`                                | Write run log to `~/execsql.log`                                |
-| `-m`                                | List metacommands and exit                                      |
-| `-n`                                | Create a new SQLite or PostgreSQL database if it does not exist |
-| `-v {0,1,2,3}`                      | GUI level (0=none, 1=password, 2=selection, 3=full)             |
-| `-w`                                | Skip password prompt when a username is supplied                |
-| `--dsn URL`                         | Connection string (e.g. `postgresql://user:pass@host/db`)       |
-| `--output-dir DIR`                  | Default base directory for EXPORT output files                  |
-| `--dry-run`                         | Parse the script and report commands without executing          |
-| `--lint`                            | Static analysis: check structure and warn on issues (no DB)     |
-| `--parse-tree`                      | Print the script's AST structure and exit (no DB)               |
-| `--list-plugins`                    | List discovered plugins and exit                                |
-| `--ping`                            | Test database connectivity and exit                             |
-| `--profile`                         | Show per-statement timing summary after execution               |
-| `--progress`                        | Show a progress bar for long-running IMPORT operations          |
-| `--config FILE`                     | Load an explicit config file (highest priority after CLI args)  |
-| `--debug`                           | Start in step-through debug mode (REPL pauses before each stmt) |
-| `--dump-keywords`                   | Print metacommand keywords as JSON and exit                     |
-| `--gui-framework {tkinter,textual}` | GUI framework for interactive prompts                           |
+| Flag                                  | Description                                                      |
+| ------------------------------------- | ---------------------------------------------------------------- |
+| `-t {p,m,s,l,k,a,f,o,d}`              | Database type                                                    |
+| `-u USER`                             | Database username                                                |
+| `-p PORT`                             | Server port                                                      |
+| `-a VALUE`                            | Set substitution variable `$ARG_x`                               |
+| `-b` / `--boolean-int`                | Treat integers 0 and 1 as boolean values                         |
+| `-c SCRIPT`                           | Execute inline SQL or metacommand string                         |
+| `-d`                                  | Auto-create export directories                                   |
+| `-e ENCODING` / `--database-encoding` | Character encoding used in the database                          |
+| `-f ENCODING`                         | Script file encoding (default: UTF-8)                            |
+| `-g ENCODING` / `--output-encoding`   | Encoding for WRITE and EXPORT output                             |
+| `-i ENCODING` / `--import-encoding`   | Encoding for data files used with IMPORT                         |
+| `-l`                                  | Write run log to `~/execsql.log`                                 |
+| `-m`                                  | List metacommands and exit                                       |
+| `-n`                                  | Create a new SQLite or PostgreSQL database if it does not exist  |
+| `-o` / `--online-help`                | Open the online documentation in the default browser             |
+| `-s N` / `--scan-lines`               | Lines to scan for IMPORT format detection (0 = scan entire file) |
+| `-v {0,1,2,3}`                        | GUI level (0=none, 1=password, 2=selection, 3=full)              |
+| `-w`                                  | Skip password prompt when a username is supplied                 |
+| `-y` / `--encodings`                  | List available encoding names and exit                           |
+| `-z KB` / `--import-buffer`           | Import buffer size in KB (default: 32)                           |
+| `--dsn URL`                           | Connection string (e.g. `postgresql://user:pass@host/db`)        |
+| `--output-dir DIR`                    | Default base directory for EXPORT output files                   |
+| `--dry-run`                           | Parse the script and report commands without executing           |
+| `--lint`                              | Static analysis: check structure and warn on issues (no DB)      |
+| `--parse-tree`                        | Print the script's AST structure and exit (no DB)                |
+| `--list-plugins`                      | List discovered plugins and exit                                 |
+| `--ping`                              | Test database connectivity and exit                              |
+| `--profile`                           | Show per-statement timing summary after execution                |
+| `--progress`                          | Show a progress bar for long-running IMPORT operations           |
+| `--config FILE`                       | Load an explicit config file (highest priority after CLI args)   |
+| `--debug`                             | Start in step-through debug mode (REPL pauses before each stmt)  |
+| `--dump-keywords`                     | Print metacommand keywords as JSON and exit                      |
+| `--gui-framework {tkinter,textual}`   | GUI framework for interactive prompts                            |
 Run `execsql --help` for the full option list, or `execsql -m` to list all metacommands.

{execsql2-2.16.0 → execsql2-2.16.1}/docs/about/contributors.md RENAMED Viewed

@@ -1,4 +1,7 @@
 # Contributors
+R. Dreas Nielsen
+:   Original author of [execsql](https://execsql.readthedocs.io/) (v1.0 through v1.130.1). Designed and implemented the metacommand language, substitution variable system, database adapters, export/import framework, GUI prompt system, and all core functionality that execsql2 is built on.
 Elizabeth Shea
 :   Brainstorming, testing, coding of the conditional expression parser, coding of the 'with arguments' and 'with parameters' clauses of the [SCRIPT](../reference/metacommands.md#beginscript) metacommand, coding of the WHILE and UNTIL clauses of the [EXECUTE SCRIPT](../reference/metacommands.md#executescript) metacommand, coding of [deferred variable substitution](../reference/substitution_vars.md#deferred_substitution), corrections to the [PROMPT ENTRY_FORM](../reference/metacommands.md#prompt_entry) metacommand, examples, implementation of the ["+" prefix](../reference/substitution_vars.md#outer_scope_referent) for substitution variables, and other minor corrections.

{execsql2-2.16.0 → execsql2-2.16.1}/docs/about/divergence.md RENAMED Viewed

@@ -185,7 +185,7 @@ print(result.variables)     # final substitution variable state
 - **DSN or connection** — pass a DSN string (`dsn="sqlite:///my.db"`) or a pre-existing `Database` object (`connection=conn`).
 - **Substitution variables** — pass a `variables` dict; keys are automatically `$`-prefixed.
 - **Error control** — `halt_on_error=True` (default) stops on the first error; `halt_on_error=False` captures errors and continues.
-- **Isolation** — each `run()` call uses an isolated `RuntimeContext`. Multiple calls do not share state.
+- **Isolation** — each `run()` call uses an isolated `RuntimeContext` stored in thread-local storage. Multiple calls do not share state, and concurrent calls from different threads are safe.
 - **Result object** — `ScriptResult` is a frozen dataclass with `success`, `commands_run`, `elapsed`, `errors`, and `variables`.
 - **Exception convenience** — call `result.raise_on_error()` to raise `ExecSqlError` if the script failed.
@@ -197,13 +197,17 @@ ______________________________________________________________________
 The CLI framework changed from `optparse` to [Typer](https://typer.tiangolo.com/) with Rich-formatted help text. All original short flags (`-a` through `-z`) are preserved. The tool can be invoked as either `execsql` or `execsql2`.
+### Default Database Type
+The default database type (`-t`) changed from Access (`a`) to SQLite (`l`). Upstream defaulted to Access, which requires Windows and pyodbc. SQLite is cross-platform, ships with Python, and is the most common use case. Users targeting Access databases should pass `-t a` explicitly.
 ### Configuration
 - **`linux_config_file`** — now only active on Linux (`sys.platform == "linux"`). Upstream applied it to all POSIX systems, including macOS. A new `macos_config_file` option handles macOS specifically.
 ### Internal State Management
-All 33 mutable runtime globals in `state.py` have been consolidated into a `RuntimeContext` object. The module uses a transparent proxy so existing code is unaffected, but the architecture now supports isolated contexts for testing and future concurrent execution.
+All 33 mutable runtime globals in `state.py` have been consolidated into a `RuntimeContext` object stored in `threading.local()`. The module uses a transparent proxy so existing code is unaffected. Each thread gets its own isolated context, enabling concurrent `from execsql import run` calls and future PARALLEL blocks.
 ### Substitution Variables
@@ -276,37 +280,38 @@ These are behavioral changes driven by security or correctness issues in the ups
 ### Bug Fixes
-| Area                                                | Fix                                                                                                                                                                                                                                                                                                    |
-| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| Oracle default port                                 | Corrected from `5432` (PostgreSQL) to `1521`.                                                                                                                                                                                                                                                          |
-| MySQL `LOAD DATA INFILE` encoding                   | Python encoding names are now mapped to MySQL charset names.                                                                                                                                                                                                                                           |
-| `dt_cast` type converters                           | Base `Database` class auto-populates 8 type converters that were previously left empty after the refactor.                                                                                                                                                                                             |
-| `FileWriter` CPU busy-loop                          | Uses blocking `queue.get(timeout=0.1)` instead of `get_nowait()` in a tight loop.                                                                                                                                                                                                                      |
-| Substitution variable cycles                        | 100-iteration limit prevents infinite loops on cyclic variable references.                                                                                                                                                                                                                             |
-| Script location in error messages                   | `ErrInfo.script_file` and `script_line_no` are now populated via `stamp_errinfo()` so error output includes "Line N of script foo.sql" context — restoring behavior present in the monolith.                                                                                                           |
-| `$ERROR_MESSAGE` not updated                        | `$ERROR_MESSAGE` is now set on every error path: `exit_now()`, non-halting SQL errors, and non-halting metacommand errors. Previously it was initialized to `""` and never changed.                                                                                                                    |
-| Metacommand error message lost                      | When `halt_on_metacommand_err` is `ON`, the original handler `ErrInfo` is now re-raised; the generic "Unknown metacommand" message no longer replaces the specific error from the handler.                                                                                                             |
-| Empty script name in error msg                      | `_execute_script_direct()` and `_execute_script_textual_console()` no longer append "in script , line 0" to uncaught-exception messages when `current_script_line()` returns an empty string.                                                                                                          |
-| `PROMPT COMPARE` diff comparison                    | Diff engine uses native Python equality instead of string comparison — numeric types, Decimals, and booleans compare correctly. `None` is distinguished from empty string. Columns are matched by name (not position), key columns are excluded from comparison, and duplicate PKs keep the first row. |
-| `win_config_file` broken                            | Checked `os.name == "windows"` which Python never returns (correct value is `"nt"`). Fixed to `os.name == "nt"`. Inherited from upstream.                                                                                                                                                              |
-| `NumericParser` right-associative                   | Arithmetic operators were parsed right-to-left. `10 - 3 - 2` evaluated as `9` instead of `5`. Fixed to left-associative parsing. Inherited from upstream.                                                                                                                                              |
-| Empty-column check precedence                       | `DataTable` and `Database.populate_table()` had an operator precedence bug in the extra-column emptiness check — a redundant `and conf.del_empty_cols` caused incorrect short-circuit evaluation. Inherited from upstream.                                                                             |
-| SQLite import string processing                     | `SQLiteDatabase.populate_table()` applied `trim_strings`, `replace_newlines`, and `empty_strings` after copying row data, so processing never reached the INSERT. Fixed to process before extraction. Inherited from upstream.                                                                         |
-| `$CURRENT_DATABASE`/`$CURRENT_DBMS` stale after USE | These variables were only set at startup and on CONNECT, not refreshed when `USE` switched the active database. Now set in `set_static_system_vars()` so they update on any connection change.                                                                                                         |
-| `DT_Text.data_type_name` wrong                      | Was `"character"` (same as `DT_Character`), making error messages indistinguishable. Corrected to `"text"`. Inherited from upstream.                                                                                                                                                                   |
-| `DT_Varchar` non-string data unchecked              | `_from_data()` only enforced the 255-char limit for `str` inputs; non-string values passed through without conversion or length check. Inherited from upstream.                                                                                                                                        |
-| `WriteHooks.write_err()` crash on empty string      | `strval[-1]` raised `IndexError` on empty input. Fixed to use `str.endswith()`. Inherited from upstream.                                                                                                                                                                                               |
-| `NumericParser` division by zero                    | `NumericAstNode.eval()` raised unhandled `ZeroDivisionError`. Now raises `NumericParserError` with a clear message. Inherited from upstream.                                                                                                                                                           |
-| `CondAstNode.eval()` could return `None`            | Missing fallthrough for unknown node types silently returned `None`. Now raises `CondParserError`. Inherited from upstream.                                                                                                                                                                            |
-| `DT_Timestamp` claims time-only values              | `dateutil.parser.parse()` silently fills in today's date for bare time strings like `"13:15:45"`, so `DT_Timestamp` matched before `DT_Time` in the inference order. `parse_datetime()` now rejects time-only strings. Inherited from upstream.                                                        |
-| `CounterVars.substitute` skipped pos 0–1            | `re.I` was passed as the positional `pos` argument, skipping the first 2 characters of every string during counter variable expansion. Counter variables at the very start of a line were never matched. Inherited from upstream.                                                                      |
-| `exec_cmd` (SQLite/DuckDB) always raised TypeError  | Passed bytes to `curs.execute()` via `.encode()`, which Python 3 `sqlite3`/`duckdb` reject. `EXECUTE PROCEDURE` metacommand was non-functional on these backends. Inherited from upstream.                                                                                                             |
-| MySQL `LOAD DATA INFILE` injection                  | File path, delimiter, and quotechar were interpolated without escaping. Now single-quotes are escaped consistent with the PostgreSQL COPY path. Inherited from upstream.                                                                                                                               |
-| Config file chain infinite loop                     | The `config_file` directive could chain config files without limit. A circular reference (via symlinks or different relative paths) caused an infinite loop at startup. Now capped at 20 files. Inherited from upstream.                                                                               |
-| Cursor leaks in database adapters                   | ~15 methods across all adapters used `curs = self.cursor()` / `curs.close()` without `try/finally`. If the query raised, the cursor leaked. Converted to `with self._cursor() as curs:`. Inherited from upstream.                                                                                      |
-| JSON export malformed on special column names       | Column names containing `"` or `\` produced invalid JSON. Now uses `json.dumps()` for all field names. Inherited from upstream.                                                                                                                                                                        |
-| Temp file creation TOCTOU race                      | `TempFileMgr.new_temp_fn()` discarded the `NamedTemporaryFile` handle, creating a race window. Now uses `tempfile.mkstemp()` for secure creation. Inherited from upstream.                                                                                                                             |
-| `shlex.split` on Windows incorrect mode             | Called without `posix=False` on Windows, mishandling backslash-heavy paths in SHELL commands. Inherited from upstream.                                                                                                                                                                                 |
+| Area                                                | Fix                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+| --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Oracle default port                                 | Corrected from `5432` (PostgreSQL) to `1521`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+| MySQL `LOAD DATA INFILE` encoding                   | Python encoding names are now mapped to MySQL charset names.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
+| `dt_cast` type converters                           | Base `Database` class auto-populates 8 type converters that were previously left empty after the refactor.                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
+| `FileWriter` CPU busy-loop                          | Uses blocking `queue.get(timeout=0.1)` instead of `get_nowait()` in a tight loop.                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
+| Substitution variable cycles                        | 100-iteration limit prevents infinite loops on cyclic variable references.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
+| Script location in error messages                   | `ErrInfo.script_file` and `script_line_no` are now populated via `stamp_errinfo()` so error output includes "Line N of script foo.sql" context — restoring behavior present in the monolith.                                                                                                                                                                                                                                                                                                                                                               |
+| `$ERROR_MESSAGE` not updated                        | `$ERROR_MESSAGE` is now set on every error path: `exit_now()`, non-halting SQL errors, and non-halting metacommand errors. Previously it was initialized to `""` and never changed.                                                                                                                                                                                                                                                                                                                                                                        |
+| Metacommand error message lost                      | When `halt_on_metacommand_err` is `ON`, the original handler `ErrInfo` is now re-raised; the generic "Unknown metacommand" message no longer replaces the specific error from the handler.                                                                                                                                                                                                                                                                                                                                                                 |
+| Empty script name in error msg                      | `_execute_script_direct()` and `_execute_script_textual_console()` no longer append "in script , line 0" to uncaught-exception messages when `current_script_line()` returns an empty string.                                                                                                                                                                                                                                                                                                                                                              |
+| `PROMPT COMPARE` diff comparison                    | Diff engine uses native Python equality instead of string comparison — numeric types, Decimals, and booleans compare correctly. `None` is distinguished from empty string. Columns are matched by name (not position), key columns are excluded from comparison, and duplicate PKs keep the first row.                                                                                                                                                                                                                                                     |
+| `win_config_file` broken                            | Checked `os.name == "windows"` which Python never returns (correct value is `"nt"`). Fixed to `os.name == "nt"`. Inherited from upstream.                                                                                                                                                                                                                                                                                                                                                                                                                  |
+| `NumericParser` right-associative                   | Arithmetic operators were parsed right-to-left. `10 - 3 - 2` evaluated as `9` instead of `5`. Fixed to left-associative parsing. Inherited from upstream.                                                                                                                                                                                                                                                                                                                                                                                                  |
+| Empty-column check precedence                       | `DataTable` and `Database.populate_table()` had an operator precedence bug in the extra-column emptiness check — a redundant `and conf.del_empty_cols` caused incorrect short-circuit evaluation. Inherited from upstream.                                                                                                                                                                                                                                                                                                                                 |
+| SQLite import string processing                     | `SQLiteDatabase.populate_table()` applied `trim_strings`, `replace_newlines`, and `empty_strings` after copying row data, so processing never reached the INSERT. Fixed to process before extraction. Inherited from upstream.                                                                                                                                                                                                                                                                                                                             |
+| `$CURRENT_DATABASE`/`$CURRENT_DBMS` stale after USE | These variables were only set at startup and on CONNECT, not refreshed when `USE` switched the active database. Now set in `set_static_system_vars()` so they update on any connection change.                                                                                                                                                                                                                                                                                                                                                             |
+| `DT_Text.data_type_name` wrong                      | Was `"character"` (same as `DT_Character`), making error messages indistinguishable. Corrected to `"text"`. Inherited from upstream.                                                                                                                                                                                                                                                                                                                                                                                                                       |
+| `DT_Varchar` non-string data unchecked              | `_from_data()` only enforced the 255-char limit for `str` inputs; non-string values passed through without conversion or length check. Inherited from upstream.                                                                                                                                                                                                                                                                                                                                                                                            |
+| `WriteHooks.write_err()` crash on empty string      | `strval[-1]` raised `IndexError` on empty input. Fixed to use `str.endswith()`. Inherited from upstream.                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
+| `NumericParser` division by zero                    | `NumericAstNode.eval()` raised unhandled `ZeroDivisionError`. Now raises `NumericParserError` with a clear message. Inherited from upstream.                                                                                                                                                                                                                                                                                                                                                                                                               |
+| `CondAstNode.eval()` could return `None`            | Missing fallthrough for unknown node types silently returned `None`. Now raises `CondParserError`. Inherited from upstream.                                                                                                                                                                                                                                                                                                                                                                                                                                |
+| `DT_Timestamp` claims time-only values              | `dateutil.parser.parse()` silently fills in today's date for bare time strings like `"13:15:45"`, so `DT_Timestamp` matched before `DT_Time` in the inference order. `parse_datetime()` now rejects time-only strings. Inherited from upstream.                                                                                                                                                                                                                                                                                                            |
+| `CounterVars.substitute` skipped pos 0–1            | `re.I` was passed as the positional `pos` argument, skipping the first 2 characters of every string during counter variable expansion. Counter variables at the very start of a line were never matched. Inherited from upstream.                                                                                                                                                                                                                                                                                                                          |
+| `exec_cmd` (SQLite/DuckDB) always raised TypeError  | Passed bytes to `curs.execute()` via `.encode()`, which Python 3 `sqlite3`/`duckdb` reject. `EXECUTE PROCEDURE` metacommand was non-functional on these backends. Inherited from upstream.                                                                                                                                                                                                                                                                                                                                                                 |
+| MySQL `LOAD DATA INFILE` injection                  | File path, delimiter, and quotechar were interpolated without escaping. Now single-quotes are escaped consistent with the PostgreSQL COPY path. Inherited from upstream.                                                                                                                                                                                                                                                                                                                                                                                   |
+| Config file chain infinite loop                     | The `config_file` directive could chain config files without limit. A circular reference (via symlinks or different relative paths) caused an infinite loop at startup. Now capped at 20 files. Inherited from upstream.                                                                                                                                                                                                                                                                                                                                   |
+| Cursor leaks in database adapters                   | ~15 methods across all adapters used `curs = self.cursor()` / `curs.close()` without `try/finally`. If the query raised, the cursor leaked. Converted to `with self._cursor() as curs:`. Inherited from upstream.                                                                                                                                                                                                                                                                                                                                          |
+| JSON export malformed on special column names       | Column names containing `"` or `\` produced invalid JSON. Now uses `json.dumps()` for all field names. Inherited from upstream.                                                                                                                                                                                                                                                                                                                                                                                                                            |
+| Temp file creation TOCTOU race                      | `TempFileMgr.new_temp_fn()` discarded the `NamedTemporaryFile` handle, creating a race window. Now uses `tempfile.mkstemp()` for secure creation. Inherited from upstream.                                                                                                                                                                                                                                                                                                                                                                                 |
+| `shlex.split` on Windows incorrect mode             | Called without `posix=False` on Windows, mishandling backslash-heavy paths in SHELL commands. Inherited from upstream.                                                                                                                                                                                                                                                                                                                                                                                                                                     |
+| AST executor `~`/`+` variable scoping broken        | The AST executor passed `localvars` through function parameters but never pushed `CommandList` frames onto `commandliststack`. Legacy metacommand handlers (`x_sub`, `x_rm_sub`, `xf_sub_defined`, `SUB_LOCAL`, prompt handlers, REPL) access `commandliststack[-1]` for `~` local and `+` outer-scope variables. This caused `~` vars to be invisible to SQL, `SUB_DEFINED(~var)` to always return false, and the REPL `.vars`/`.stack` to show empty state. Fixed by pushing/popping `CommandList` frames in `execute()` and `_execute_script_native()`. |
 ______________________________________________________________________

{execsql2-2.16.0 → execsql2-2.16.1}/docs/api/exporters.md RENAMED Viewed

@@ -28,10 +28,16 @@ Each module below implements one or more output formats. Every writer follows th
 ::: execsql.exporters.latex
+::: execsql.exporters.markdown
 ::: execsql.exporters.ods
 ::: execsql.exporters.xls
+::: execsql.exporters.xlsx
+::: execsql.exporters.yaml
 ::: execsql.exporters.feather
 ::: execsql.exporters.parquet

{execsql2-2.16.0 → execsql2-2.16.1}/docs/api/importers.md RENAMED Viewed

@@ -14,6 +14,8 @@ Each module below implements one or more import formats.
 ::: execsql.importers.csv
+::: execsql.importers.json
 ::: execsql.importers.ods
 ::: execsql.importers.xls

{execsql2-2.16.0 → execsql2-2.16.1}/docs/api/index.md RENAMED Viewed

@@ -24,6 +24,30 @@ result: ScriptResult = run(
 See the [README](https://github.com/geocoug/execsql#library-api) for full examples.
+### Thread Safety
+`run()` is **thread-safe**. Each call creates an isolated `RuntimeContext` stored in thread-local storage, so concurrent calls from different threads do not share database connections, substitution variables, or execution state.
+```python
+import threading
+from execsql import run
+def etl_worker(script, dsn):
+    result = run(script=script, dsn=dsn)
+    print(f"{script}: {'OK' if result.success else 'FAIL'}")
+threads = [
+    threading.Thread(target=etl_worker, args=("load_us.sql", "postgresql://host/us_db")),
+    threading.Thread(target=etl_worker, args=("load_eu.sql", "postgresql://host/eu_db")),
+]
+for t in threads:
+    t.start()
+for t in threads:
+    t.join()
+```
+Each thread gets its own database connections, IF/LOOP stacks, substitution variables, and error state. No locking is required.
 ## Extension Guides
 | Extension type       | Guide                                                    | API reference                   |

{execsql2-2.16.0 → execsql2-2.16.1}/docs/dev/adding_metacommands.md RENAMED Viewed

@@ -42,16 +42,20 @@ ______________________________________________________________________
 Add the handler to whichever sibling module fits best:
-| Module          | Handles                                               |
-| --------------- | ----------------------------------------------------- |
-| `connect.py`    | Database connections, `USE`, `DISCONNECT`             |
-| `control.py`    | Control flow: `IF`, `LOOP`, `BREAK`, `HALT`, batch    |
-| `data.py`       | Variable manipulation: `SUB`, `SUBDATA`, counters     |
-| `io.py`         | File I/O: `EXPORT`, `IMPORT`, `INCLUDE`, `WRITE`      |
-| `prompt.py`     | User interaction: `PROMPT`, `ASK`, `PAUSE`, `MSG`     |
-| `system.py`     | OS interaction: `SYSTEM_CMD`, `LOG`, `EMAIL`, console |
-| `debug.py`      | Debug output                                          |
-| `script_ext.py` | Script extensions                                     |
+| Module          | Handles                                                                |
+| --------------- | ---------------------------------------------------------------------- |
+| `connect.py`    | Database connections, `USE`, `DISCONNECT`                              |
+| `control.py`    | Control flow: `IF`, `LOOP`, `BREAK`, `HALT`, batch                     |
+| `data.py`       | Variable manipulation: `SUB`, `SUBDATA`, counters                      |
+| `io.py`         | Re-export façade only -- imports and re-exports from the modules below |
+| `io_export.py`  | `EXPORT` handlers                                                      |
+| `io_import.py`  | `IMPORT` handlers                                                      |
+| `io_write.py`   | `WRITE` and `WRITESCRIPT` handlers                                     |
+| `io_fileops.py` | `INCLUDE`, `ZIP`, `CD`, `SERVE`, and other file operations             |
+| `prompt.py`     | User interaction: `PROMPT`, `ASK`, `PAUSE`, `MSG`                      |
+| `system.py`     | OS interaction: `SYSTEM_CMD`, `LOG`, `EMAIL`, console                  |
+| `debug.py`      | Debug output                                                           |
+| `script_ext.py` | Script extensions                                                      |
 If none of these fit, create a new module and follow the same structure.

{execsql2-2.16.0 → execsql2-2.16.1}/docs/dev/architecture.md RENAMED Viewed

@@ -61,17 +61,25 @@ flowchart LR
     CONFIG["config.py<br/>ConfigData,<br/>StatObj,<br/>WriteHooks"]
     STATE["state.py<br/>Global runtime<br/>singletons"]
     SCRIPT["script/<br/>engine, control,<br/>variables"]
-    META["metacommands/<br/>Dispatch table,<br/>~200 handlers"]
+    META["metacommands/<br/>Dispatch table,<br/>~225 handlers"]
     DB["db/<br/>Database ABC,<br/>9 adapters,<br/>DatabasePool"]
-    EXPORT["exporters/<br/>15+ output<br/>formats"]
+    EXPORT["exporters/<br/>20+ output<br/>formats"]
     IMPORT["importers/<br/>CSV, ODS,<br/>XLS, Feather"]
     GUI["gui/<br/>Tkinter, Textual,<br/>Console backends"]
     UTILS["utils/<br/>auth, crypto,<br/>fileio, mail,<br/>regex, strings"]
     PARSER["parser.py<br/>CondParser,<br/>NumericParser"]
+    DEBUG["debug/<br/>REPL debugger"]
+    NOTEBOOK["notebook/<br/>Jupyter integration"]
+    SERVER["server/<br/>Daemon"]
+    LSP["lsp/<br/>Language Server<br/>Protocol"]
     CLI --> CONFIG
     CLI --> STATE
     CLI --> SCRIPT
+    CLI --> DEBUG
+    CLI --> NOTEBOOK
+    CLI --> SERVER
+    CLI --> LSP
     SCRIPT --> STATE
     SCRIPT --> META
     META --> STATE
@@ -92,11 +100,11 @@ flowchart LR
 | --------------- | ------------------------------------------------------------------------------------------------ |
 | `cli/`          | Typer app, `_run()` orchestration, DSN URL parsing, Rich help output                             |
 | `config.py`     | `ConfigData` (INI merging), `StatObj` (runtime flags), `WriteHooks` (stdout/stderr redirection)  |
-| `state.py`      | Module-level global runtime store -- all shared mutable state lives here                         |
+| `state.py`      | Thread-local runtime store -- all shared mutable state lives here, isolated per-thread           |
 | `script/`       | `CommandList`, `MetaCommandList`, `SubVarSet`, `ScriptFile`, `runscripts()`                      |
 | `metacommands/` | `build_dispatch_table()`, all `x_*` handlers, `build_conditional_table()`, all `xf_*` predicates |
 | `db/`           | `Database` ABC, `DatabasePool`, 9 adapter modules (postgres, sqlite, duckdb, mysql, etc.)        |
-| `exporters/`    | `ExportRecord`, `ExportMetadata`, `WriteSpec`, 15+ format writers (CSV, JSON, XML, HTML, etc.)   |
+| `exporters/`    | `ExportRecord`, `ExportMetadata`, `WriteSpec`, 20+ format writers (CSV, JSON, XML, HTML, etc.)   |
 | `importers/`    | `CsvFile`, `OdsFile`, `XlsFile`, `FeatherFile` -- data import backends                           |
 | `gui/`          | `GuiBackend` ABC, `TkinterBackend`, `TextualBackend`, `ConsoleBackend`                           |
 | `utils/`        | Shared utilities: file I/O, encryption, mail, regex helpers, string manipulation, timers         |
@@ -105,6 +113,10 @@ flowchart LR
 | `models.py`     | `Column`, `DataTable`, `JsonDatatype`                                                            |
 | `exceptions.py` | `ExecSqlError` base, `ErrInfo`, `ConfigError`, `DataTypeError`, `DbTypeError`, etc.              |
 | `plugins.py`    | Entry-point plugin discovery for metacommands, exporters, and importers                          |
+| `debug/`        | Interactive REPL debugger for stepping through script execution                                  |
+| `notebook/`     | Jupyter notebook integration -- run execsql scripts from notebook cells                          |
+| `server/`       | Daemon mode -- run execsql as a long-lived process accepting script requests                     |
+| `lsp/`          | Language Server Protocol server -- powers editor features such as hover docs and autocomplete    |
 ______________________________________________________________________
@@ -121,7 +133,7 @@ The AST executor is the only execution engine. Use `--parse-tree` to visualize t
 ### RuntimeContext Isolation
-The executor accepts an explicit `RuntimeContext` parameter (`ctx`). The `active_context()` context manager in `state.py` installs a context as the active global so all metacommand handlers and database adapters automatically resolve against it. This enables isolated execution for the library API. The RuntimeContext carries AST-specific state: `ast_scripts` (script block registry) and `include_chain` (circular include detection).
+The executor accepts an explicit `RuntimeContext` parameter (`ctx`). The `active_context()` context manager in `state.py` installs a context as the active thread-local so all metacommand handlers and database adapters automatically resolve against it. Each thread gets its own isolated context via `threading.local()`, enabling concurrent `from execsql import run` calls and future PARALLEL blocks. The RuntimeContext carries AST-specific state: `ast_scripts` (script block registry) and `include_chain` (circular include detection).
 ______________________________________________________________________
@@ -199,12 +211,12 @@ ______________________________________________________________________
 ## Metacommand Dispatch
-Metacommands are lines in SQL scripts prefixed with `-- !x!`. At import time, `metacommands/__init__.py` calls `build_dispatch_table()`, which populates a `MetaCommandList` with all `mcl.add()` registrations (~205 regex patterns). This singleton is stored as `_state.metacommandlist`.
+Metacommands are lines in SQL scripts prefixed with `-- !x!`. At import time, `metacommands/__init__.py` calls `build_dispatch_table()`, which populates a `MetaCommandList` with all `mcl.add()` registrations (~225 regex patterns). This singleton is stored as `_state.metacommandlist`.
 ### How dispatch works
 1. `MetacommandStmt.run()` calls `_state.metacommandlist.eval(cmd_str)`.
-1. `MetaCommandList.eval()` extracts the leading keyword from the command string and looks up candidate `MetaCommand` entries via a keyword index (reducing the search space from ~205 to typically 1-5 entries).
+1. `MetaCommandList.eval()` extracts the leading keyword from the command string and looks up candidate `MetaCommand` entries via a keyword index (reducing the search space from ~225 to typically 1-5 entries).
 1. Each candidate's compiled regex is tested against the full command string.
 1. On a match, the handler function is called with all named regex groups as keyword arguments, plus `metacommandline` (the original unmodified line).
 1. On success, the matched entry's hit counter is incremented.
@@ -352,13 +364,13 @@ ______________________________________________________________________
 ## Global State
-`state.py` is the shared mutable state store for the entire runtime. All other modules access it via:
+`state.py` is the thread-local mutable state store for the runtime. All other modules access it via:
 ```python
 import execsql.state as _state
 ```
-This import pattern (always as `_state`, always accessed inside function/method bodies) avoids circular imports at load time. The module declares all globals at the top level and provides two management functions:
+This import pattern (always as `_state`, always accessed inside function/method bodies) avoids circular imports at load time. The module stores all mutable state in a `threading.local()` instance, so each thread gets its own isolated `RuntimeContext`. The module provides two management functions:
 - **`initialize(config, dispatch_table, conditional_table)`** -- Called once from `_run()` to create runtime singletons (`DatabasePool`, `IfLevels`, `CounterVars`, etc.).
 - **`reset()`** -- Tears down all state for test isolation.

execsql2 2.16.0__tar.gz → 2.16.1__tar.gz

execsql2 2.16.0tar.gz → 2.16.1tar.gz