execsql2 2.15.11__tar.gz → 2.16.0__tar.gz

execsql2-2.16.0/AUDIT.md

@@ -0,0 +1,324 @@
+# Codebase Audit — execsql2
+
+**Original audit:** 2026-04-13
+**Last updated:** 2026-04-29
+
+Full audit covered 42 source files and 12 test files. 25 findings were identified (F1-F25)
+plus 29 additional findings across security, performance, test gaps, edge cases, code quality,
+and divergences. Of those, **25 findings have been resolved** (10 bugs, 3 perf fixes, 2 edge
+cases, 1 security fix, 8 doc fixes, 1 CI fix, and the F-series fixes below).
+
+**Resolved F-series:** F1 (substitute_all tuple unpack), F2 (bytes-to-stdout), F5 (export
+dispatch duplication), F7 (commandliststack bounds), F8 (exec_cmd injection), F10 (enc_password
+deprecation warning), F12 (env var filtering), F13 (bumpversion --no-verify), F14 (dead
+\_DEFAULT_CTX), F17 (duplicate tuple entries), F20 (password persistence), F24 (Comment AST node
+implemented).
+
+**User-excluded:** F11 (--allow-shell flag) — not adding.
+
+______________________________________________________________________
+
+## Architecture & Code Quality
+
+### F3. [HIGH] Global mutable state blocks threading and library API
+
+`src/execsql/state.py:399-432` — `_ctx` is a plain module-level global. All 200+ metacommand
+handlers, SQL execution, variable substitution, and the IF-stack go through `_state.foo` which
+resolves to `_ctx.foo`. Two threads executing scripts simultaneously will corrupt each other's
+state. Blocks the planned PARALLEL metacommand and safe `from execsql import run` usage.
+
+**Fix:** Replace `_ctx` with `contextvars.ContextVar` or `threading.local()`.
+
+### F4. [HIGH] `_run()` is a 460-line god-function with 36 parameters
+
+`src/execsql/cli/run.py:193-660` — Performs config parsing, CLI argument merging, DSN parsing,
+database connection, GUI init, script loading, dry-run, AST execution, legacy execution,
+profiling, and cleanup in a single function. Estimated cyclomatic complexity ~85-90. Untestable
+as a unit.
+
+**Fix:** Extract into phases: `_setup_config()`, `_connect_db()`, `_load_script()`,
+`_execute()`, `_teardown()`.
+
+### F15. [MEDIUM] Legacy parser is a 150-line function with 6 levels of nesting
+
+`src/execsql/script/engine.py:864-1012` — The production parser tracks 6 state variables with
+deepest nesting at 6 levels. The AST parser is the planned replacement; accelerating that
+migration is the real fix.
+
+### F19. [MEDIUM] 51 near-duplicate regex patterns in conditions.py
+
+`src/execsql/metacommands/conditions.py:426-627` — `CONTAINS`, `STARTS_WITH`, `ENDS_WITH` each
+register 17 quoting variants as separate regex patterns. Adding a new string predicate requires
+copying and modifying 17 patterns.
+
+**Fix:** Consolidate into a single pattern per predicate with optional quoting alternation groups.
+
+### CQ-1. [MEDIUM] 111 bare `except Exception` clauses across 38 files
+
+Many suppress errors silently (`pass`). Notable: `db/base.py` (14), `utils/fileio.py` (9),
+`db/access.py` (10). Each should be reviewed for more specific exception types.
+
+### CQ-4. [MEDIUM] Inconsistent `_cursor()` context manager vs bare `cursor()`
+
+The `Database` base class provides `_cursor()` for cleanup, but many adapter methods still use
+`curs = self.cursor()` without cleanup. Standardize to `with self._cursor() as curs:`.
+
+### F22. [LOW] Eager import of `AccessDatabase` in connect.py
+
+`src/execsql/metacommands/connect.py:1-2` — Imports `AccessDatabase` (depends on pyodbc,
+Windows-only) at module load. Moving to lazy import was reverted because test patch targets
+broke. Marginal benefit.
+
+### F23. [LOW] `conftest.py` `minimal_conf` fixture doesn't cover all attributes
+
+`tests/conftest.py:33-56` — `SimpleNamespace` with ~16 attributes vs `ConfigData.__init__`'s
+~50+. Tests hitting unset attributes get `AttributeError`.
+
+### F25. [LOW] `format.py` SQL formatter hardcodes PostgreSQL dialect
+
+`src/execsql/format.py:155,160` — `sqlglot.parse(read="postgres")` regardless of target DB.
+
+**Fix:** Accept a `--dialect` flag or infer from `--type`.
+
+### CQ-2. [LOW] `JsonDatatype` class uses class-level attribute assignment anti-pattern
+
+`src/execsql/models.py:308-324` — `JsonDatatype.integer` assigned twice (lines 317 and 322).
+Could be an enum or dict.
+
+### CQ-3. [LOW] `Encrypt.ky` is a mutable class variable
+
+`src/execsql/utils/crypto.py:48-57` — Dict never mutated after definition but declared as
+mutable class attribute. Frozen dict or class constant would be cleaner.
+
+### CQ-5. [LOW] `state.py` version parsing catches bare `Exception`
+
+`src/execsql/state.py:148` — Should be `except (ValueError, IndexError)`.
+
+### CQ-6. [LOW] `ConfigData.export_output_dir` is dynamically added
+
+`src/execsql/cli/run.py:365` — Attribute doesn't exist in `ConfigData.__init__()`. Should be
+declared with default `None`.
+
+______________________________________________________________________
+
+## Security
+
+### SEC-1. [MEDIUM] SHELL metacommand passes user input to subprocess
+
+`src/execsql/metacommands/system.py:38-56` — `shlex.split()` tokenizes but doesn't sanitize.
+Substitution variables from `PROMPT ENTRY`, `SUB_INI`, or env vars could inject arguments.
+Partially mitigated: `subprocess.call` with list arg doesn't invoke a shell. User excluded
+`--allow-shell` flag (F11).
+
+### SEC-3. [LOW] `Encrypt` class provides no real security
+
+`src/execsql/utils/crypto.py` — Hardcoded XOR keys in source. Any `enc_password` can be
+trivially decoded. Documented as obfuscation-only. Deprecation warning now emitted (F10 fix).
+
+### SEC-4. [LOW] `x_include` path traversal
+
+`src/execsql/metacommands/io_fileops.py:26-38` — `INCLUDE` accepts arbitrary file paths from
+scripts. If variables are populated from external sources, arbitrary files could be included.
+
+______________________________________________________________________
+
+## Performance
+
+### F18. [MEDIUM] `SubVarSet._substitute_nested` is O(V) per call
+
+`src/execsql/script/variables.py:262-296` — Fallback iterates over every defined variable with
+case-insensitive substring search. With N tokens and V variables, worst case is O(N * V).
+
+**Fix:** Index variables by first few characters for faster fallback lookup.
+
+### PERF-4. [LOW] `substitute_vars` creates merged `SubVarSet` per statement
+
+`src/execsql/script/engine.py:778-783` — When `localvars` is not None, creates a new
+`SubVarSet` with recompiled regex every statement. Could cache when local vars haven't changed.
+
+### PERF-5. [LOW] `set_dynamic_system_vars()` called per-statement
+
+`src/execsql/script/engine.py:738-761` — 7 `add_substitution` calls per statement for values
+that only change on CONFIG/AUTOCOMMIT metacommands. A dirty-flag approach would avoid overhead.
+
+### PERF-6. [LOW] `date_fmts` deque is shared module-level mutable
+
+`src/execsql/types.py:184-204` — Currently harmless but would be a race condition under future
+parallelism.
+
+### PERF-7. [LOW] Pretty-print materializes entire result set
+
+`src/execsql/exporters/pretty.py:47-49` — `list(rows)` forces the entire streaming generator
+into memory to compute column widths. A 1M-row result set could blow up memory.
+
+______________________________________________________________________
+
+## Test Gaps
+
+### F6. [HIGH] Two parsers with no equivalence tests
+
+`src/execsql/script/engine.py:864-1012` (legacy) and `src/execsql/script/parser.py` (AST) —
+Two independent parsers for the same grammar with no shared test verifying equivalent results.
+The AST parser creates nested block nodes; the legacy parser treats IF/LOOP/BATCH as flat.
+
+**Fix:** Add parametric tests running a script corpus through both parsers and asserting
+structural equivalence.
+
+### F9. [HIGH] Core SQL execution unit tests are theater
+
+`tests/test_engine.py:319-412` — `SqlStmt.run()` never tested with a real DB.
+`MetacommandStmt.run()` patches the entire dispatch table. `CommandList` patches
+`run_and_increment`. Error-recovery paths (`WriteSpec.write()`, `MailSpec.send()`) still have
+zero integration test coverage.
+
+**Fix:** Unit tests for `SqlStmt.run()` with real in-memory SQLite. Integration tests for
+`ON ERROR_HALT WRITE` and `ON ERROR_HALT EMAIL` end-to-end.
+
+### F16. [MEDIUM] Coverage omissions exclude ~7.8K lines
+
+`pyproject.toml:204-222` — Excludes gui/desktop.py, gui/tui.py, metacommands/prompt.py,
+script/executor.py, all 7 DB adapters, all LSP modules. The 90% gate applies to ~27K of ~35K
+lines.
+
+### TEST-1. [MEDIUM] No tests for `debug/repl.py`
+
+REPL commands (`.vars`, `.set`, `.where`, `.stack`, ad-hoc SQL) have no dedicated tests.
+
+### TEST-5. [MEDIUM] Limited importer edge case coverage
+
+CSV importer doesn't cover: inconsistent column counts, encoding errors, BOM markers, empty
+files, header-only files.
+
+### TEST-2. [LOW] No tests for `gui/desktop.py` or `gui/tui.py` (excluded from coverage)
+
+### TEST-3. [LOW] No tests for `metacommands/prompt.py` (excluded from coverage)
+
+### TEST-4. [LOW] No tests for `db/dsn.py` (excluded from coverage)
+
+### TEST-6. [LOW] No `format.py` edge case tests
+
+Doesn't cover: nested block comments, metacommands inside block comments, SQL strings
+containing `-- !x!` patterns, very long lines.
+
+### TEST-7. [LOW] No property-based tests for parsers
+
+`CondParser` and `NumericParser` handle arbitrary user input. Hypothesis-based testing would
+catch edge cases.
+
+### TEST-8. [LOW] DB adapters have zero test coverage
+
+`db/access.py`, `db/firebird.py`, `db/oracle.py`, `db/sqlserver.py` — all excluded from
+coverage.
+
+______________________________________________________________________
+
+## Edge Cases & Robustness
+
+### EDGE-2. [LOW] `SubVarSet.substitute_all()` has no cycle depth limit
+
+`src/execsql/script/variables.py:254-265` — `substitute_vars()` in engine.py has a 100-iteration
+guard, but `substitute_all()` called directly (e.g., from config loading) has no guard.
+
+### EDGE-3. [LOW] `ScriptFile.__repr__` uses `super().filename` incorrectly
+
+`src/execsql/script/engine.py:635` — Should be `self.filename`.
+
+### EDGE-5. [LOW] Block comment parsing doesn't handle nested comments
+
+`src/execsql/script/engine.py:871-881` — Simple `in_block_cmt` flag; `/* outer /* inner */ still comment */` exits at first `*/`.
+
+### EDGE-6. [LOW] `SourceString.match_str()` parameter shadows builtin `str`
+
+`src/execsql/parser.py:60` — Suppressed by ruff A002.
+
+### EDGE-7. [LOW] `_import_loop` may access unbound `line` variable
+
+`src/execsql/db/base.py:565` — If `StopIteration` raised on first row, `line` would be
+unbound. Guarded in practice by `len(b) > 0`.
+
+______________________________________________________________________
+
+## Divergences from Monolith
+
+### DIV-1. [LOW] `ScriptCmd` resolves `source_dir` at construction time
+
+`src/execsql/script/engine.py:400-408` — Monolith resolved per-statement. New behavior is
+arguably better. Not documented in `divergence.md`.
+
+### DIV-2. [MEDIUM] `$CURRENT_DATABASE`/`$CURRENT_DBMS` set differently
+
+Static (per-connect) vs dynamic (per-statement) split means these aren't updated on `USE`.
+
+### DIV-3. [LOW] `DT_Long` maps to "hugeint" in SQLite
+
+`src/execsql/types.py:742` — SQLite doesn't have "hugeint"; gets TEXT affinity.
+
+### DIV-4. [LOW] `DT_DuckDB` character types all map to TEXT
+
+`src/execsql/types.py:761-763` — Loses VARCHAR(N) length constraints.
+
+### DIV-5. [LOW] Keyring stores password silently in GUI mode
+
+`src/execsql/utils/auth.py:192-193` — Auto-stores without asking. Divergence doc mentions
+keyring but not auto-store behavior.
+
+______________________________________________________________________
+
+## Residual Risks
+
+1. **Thread safety (F3)** is the largest residual risk. `from execsql import run` cannot be
+    used from multiple threads. Must be addressed before the library API is promoted.
+
+1. **Env var filter (F12 fix)** is a behavioral change — scripts relying on
+    `!!&AWS_SECRET_ACCESS_KEY!!` will silently get empty strings. No opt-out mechanism.
+
+1. **exec_cmd quoting (F8 fix)** — `quote_identifier("schema.myproc")` produces
+    `"schema.myproc"` (single identifier), not `"schema"."myproc"`. No worse than before but
+    could break schema-qualified function calls.
+
+1. **WriteSpec/MailSpec** error-recovery paths are now correct but still have zero dedicated
+    integration tests.
+
+1. **Bumpversion hook removal (F13 fix)** — if pre-commit hooks reject bump-generated content,
+    bumps will fail. Watch on next version bump.
+
+______________________________________________________________________
+
+## Feature Ideas
+
+### Quick wins
+
+- `$TIMER_SECONDS` — numeric companion to `$TIMER` timedelta string
+- `$CURRENT_DATE` — clean `YYYY-MM-DD` string (vs `$DATE_TAG`'s `YYYYMMDD`)
+- `CONTINUE` in loops — only `BREAK` exists; users nest IF blocks as workaround
+
+### Medium features
+
+- `FOR <var> IN <query|list>` loop — avoids `SUBDATA` + `WHILE` + string manipulation
+- `RETRY N [BACKOFF s]` — transient DB error handling without verbose manual patterns
+- Native webhook/HTTP notifications — `ON ERROR_HALT WEBHOOK` instead of `SYSTEM_CMD curl`
+- `IMPORT FROM URL` — HTTP/REST import without temp file management
+- Entry form validation enforcement — `validation_regex` fields exist but aren't enforced
+
+### Strategic features
+
+- Textual TUI console — `CONSOLE ON` is a stub in the Textual backend
+- Persistent state across runs — `~/.execsql/state.db` for run history, watermarks, checkpoints
+- Thread-safe RuntimeContext (F3) — enables PARALLEL blocks, concurrent library API, easier testing
+- AST migration completion (F6/F15) — foundational for LSP and long-term maintainability
+- Plugin system via entry points — custom metacommands, community ecosystem
+- LSP enhancements — autocomplete, hover docs, jump-to-definition, inline diagnostics
+
+______________________________________________________________________
+
+## Summary
+
+| Category               | High  | Medium | Low    | Total  |
+| ---------------------- | ----- | ------ | ------ | ------ |
+| Architecture & Quality | 2     | 2      | 6      | 10     |
+| Security               | 0     | 1      | 2      | 3      |
+| Performance            | 0     | 1      | 4      | 5      |
+| Test Gaps              | 2     | 3      | 6      | 11     |
+| Edge Cases             | 0     | 0      | 5      | 5      |
+| Divergences            | 0     | 1      | 4      | 5      |
+| **Total**              | **4** | **8**  | **27** | **39** |

{execsql2-2.15.11 → execsql2-2.16.0}/CHANGELOG.md

@@ -13,6 +13,56 @@ ______________________________________________________________________
 
 ______________________________________________________________________
 
+## [2.16.0] - 2026-04-29
+
+### Added
+
+- `--parse-tree` CLI flag: parse a script into an Abstract Syntax Tree and print a visual tree structure showing block nesting (IF/LOOP/BATCH/SCRIPT), source line ranges, compound conditions (ANDIF/ORIF), and all metacommands. Requires no database connection or configuration.
+- AST parser module (`execsql.script.parser`) with `parse_script()` and `parse_string()` entry points. Produces a structured `Script` tree with typed nodes for all block constructs (IfBlock, LoopBlock, BatchBlock, ScriptBlock, SqlBlock, IncludeDirective).
+- AST node definitions (`execsql.script.ast`) with `format_tree()` for human-readable tree output.
+- AST-based execution engine is now the default (and only) engine. Scripts are parsed into a tree of typed nodes, then walked for execution. INCLUDE'd files are parsed and executed natively with circular-include detection. Control flow (IF/LOOP/BATCH) is driven by tree structure.
+- `active_context()` context manager in `execsql.state` for installing an isolated `RuntimeContext` as the active global context within a `with` block.
+- Plugin system (`execsql.plugins`) for extending execsql with custom metacommands, export formats, and import formats via Python entry points. Entry point groups: `execsql.metacommands`, `execsql.exporters`, `execsql.importers`. Plugins are discovered automatically at startup.
+- `--list-plugins` CLI flag to show all discovered plugins and exit.
+- Python library API: `from execsql import run` for programmatic script execution from notebooks, pipelines, and applications. Returns a `ScriptResult` with success/failure, command count, timing, errors, and final variable state. Supports DSN connection strings, pre-existing connections, substitution variables, and error control. Full RuntimeContext isolation between calls.
+- AST `Comment` node: the parser now preserves SQL comments in the tree. Consecutive single-line `--` comments are grouped into one node; `/* */` block comments are captured as single nodes. The `--parse-tree` output includes `<CMT>` tagged comment nodes.
+- `--parse-tree` visual improvements: color-coded type tags (`<SQL>`, `<CMD>`, `<CMT>`, `<IF>`, `<LOOP>`, etc.), dimmed line numbers, and content truncation for cleaner output.
+- Deprecation warning emitted when `enc_password` is used in config files, advising users to switch to keyring or environment variables.
+- Sensitive environment variables (`*SECRET*`, `*TOKEN*`, `*PASSWORD*`, etc.) are now filtered from automatic substitution variable exposure.
+
+### Changed
+
+- **Execution engine replaced.** The legacy flat command-list engine has been replaced by the AST-based executor. Scripts are now parsed into a tree of typed nodes and executed by walking the tree. INCLUDE'd files are parsed and executed natively with circular-include detection. All metacommands, SQL, and control flow work identically. This change is transparent to users.
+- **BREAK outside LOOP is now an error.** `BREAK` outside a loop block now raises an error (exit 1) instead of being silently ignored. This catches script bugs that were previously unreported.
+- `--lint` now uses the AST parser for structural validation. Unmatched IF/LOOP/BATCH/SCRIPT blocks are caught at parse time with precise source line ranges. No database connection or runtime state initialization is required. All prior lint checks (variable analysis, INCLUDE file existence, EXECUTE SCRIPT resolution, SUB_INI reading) are preserved.
+- Export format dispatch logic (`EXPORT` and `EXPORT QUERY` metacommands) refactored from duplicated ~180-line if/elif chains into shared `_dispatch_format()` function, eliminating code duplication and fixing missing zip-compatibility checks for `EXPORT QUERY`.
+- `MailSpec.send()` refactored: extracted `_expand()` helper to replace 12 repetitive substitution lines.
+
+### Fixed
+
+- **[Critical]** `WriteSpec.write()` and `MailSpec.send()` error-recovery paths crashed because `SubVarSet.substitute_all()` returns `(str, bool)` but callers treated the return as a plain string. All 14 call sites now unpack the tuple correctly.
+- **[Critical]** Error-recovery fallback in `WriteSpec.write()` and `io_write` called `.encode()` producing bytes passed to `sys.stdout.write()` which expects `str`. Removed the `.encode()` calls.
+- `WriteSpec.write()` no longer crashes with `IndexError` when `commandliststack` is empty during early initialization errors.
+- SQL injection vector in `exec_cmd()` across all 8 database adapters — stored procedure/function/view names are now quoted with `quote_identifier()`.
+- `DSN` and `SQL Server` adapters no longer encode SQL strings to bytes before execution.
+- Duplicate tuple entries in export format checks (`"txt-and"` and `"text-and"` each appeared twice).
+- Database adapters now clear `self.password` after successful connection, reducing credential exposure window.
+- Removed unused `_DEFAULT_CTX = RuntimeContext()` allocation in `state.py`.
+- Version bump commits no longer skip pre-commit hooks (`--no-verify` removed from bumpversion config).
+- `SubVarSet.substitute_all()` now enforces a 100-iteration depth limit to prevent infinite loops from cyclic variable references. The per-statement guard in the executor already had this protection, but direct callers (e.g. config loading) did not.
+- `ConfigData.export_output_dir` is now declared in `__init__` with a default of `None` instead of being dynamically added in the CLI entry point.
+- `Encrypt.ky` key table is now an immutable `MappingProxyType` instead of a mutable class-level dict.
+- `JsonDatatype` attributes are now declared as class variables in the class body instead of assigned externally after class definition.
+- `minimal_conf` test fixture expanded with commonly needed attributes (`import_encoding`, `script_encoding`, `export_output_dir`, `write_prefix`, `write_suffix`, `fold_col_hdrs`, `trim_col_hdrs`, etc.) to reduce ad-hoc attribute additions in individual tests.
+
+### Removed
+
+- `--ast` / `--no-ast` CLI flag — the AST executor is now the only execution engine; no opt-out.
+- Legacy flat command-list execution engine (`_parse_script_lines`, `read_sqlfile`, `read_sqlstring`, `runscripts`, `ScriptFile`, `CommandListWhileLoop`, `CommandListUntilLoop`, `ScriptExecSpec.execute()`).
+- Legacy `_execute_script_direct()` function and `_execute_include_legacy()` fallback path.
+
+______________________________________________________________________
+
 ## [2.15.11] - 2026-04-27
 
 ### Fixed

{execsql2-2.15.11 → execsql2-2.16.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: execsql2
-Version: 2.15.11
+Version: 2.16.0
 Summary: Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables.
 Project-URL: Homepage, https://execsql2.readthedocs.io
 Project-URL: Repository, https://github.com/geocoug/execsql
@@ -237,6 +237,8 @@ execsql script.sql                          # read connection from config file
 | `--output-dir DIR`                  | Default base directory for EXPORT output files                  |
 | `--dry-run`                         | Parse the script and report commands without executing          |
 | `--lint`                            | Static analysis: check structure and warn on issues (no DB)     |
+| `--parse-tree`                      | Print the script's AST structure and exit (no DB)               |
+| `--list-plugins`                    | List discovered plugins and exit                                |
 | `--ping`                            | Test database connectivity and exit                             |
 | `--profile`                         | Show per-statement timing summary after execution               |
 | `--progress`                        | Show a progress bar for long-running IMPORT operations          |
@@ -260,6 +262,61 @@ Run `execsql --help` for the full option list, or `execsql -m` to list all metac
 - Display query results in a GUI dialog; optionally prompt the user to select a row, enter a value, or submit a form.
 - Write status messages or tabular output to the console or a file during execution.
 - Automatically log each run, recording databases used, scripts executed, and user responses.
+- Extend with custom metacommands, exporters, and importers via the plugin system.
+
+# Library API
+
+execsql can be used as a Python library for programmatic script execution:
+
+```python
+from execsql import run
+
+# Execute a script file
+result = run(script="pipeline.sql", dsn="postgresql://user:pass@host/db")
+
+# Execute inline SQL
+result = run(
+    sql="CREATE TABLE t (id INT);\nINSERT INTO t VALUES (1);",
+    dsn="sqlite:///my.db",
+    new_db=True,
+)
+
+# With substitution variables
+result = run(
+    script="etl.sql",
+    dsn="sqlite:///data.db",
+    variables={"SCHEMA": "public", "DATE": "2026-01-01"},
+)
+
+# Check results
+print(result.success)       # True
+print(result.commands_run)  # 2
+print(result.elapsed)       # 0.003 (seconds)
+print(result.variables)     # {"SCHEMA": "public", ...}
+```
+
+Error handling:
+
+```python
+result = run(sql="SELECT * FROM nonexistent;", dsn="sqlite:///:memory:")
+if not result.success:
+    for err in result.errors:
+        print(f"{err.source}:{err.line}: {err.message}")
+
+# Or raise on failure
+result.raise_on_error()  # raises ExecSqlError
+```
+
+Use a pre-existing database connection instead of a DSN:
+
+```python
+from execsql.db.factory import db_SQLite
+conn = db_SQLite("my.db", new_db=True)
+result = run(sql="SELECT 1;", connection=conn)
+# run() does NOT close this connection — you manage its lifecycle
+```
+
+Each call to `run()` uses an isolated `RuntimeContext`, so multiple calls do not share state.
 
 # An Illustration
 

{execsql2-2.15.11 → execsql2-2.16.0}/README.md

@@ -115,6 +115,8 @@ execsql script.sql                          # read connection from config file
 | `--output-dir DIR`                  | Default base directory for EXPORT output files                  |
 | `--dry-run`                         | Parse the script and report commands without executing          |
 | `--lint`                            | Static analysis: check structure and warn on issues (no DB)     |
+| `--parse-tree`                      | Print the script's AST structure and exit (no DB)               |
+| `--list-plugins`                    | List discovered plugins and exit                                |
 | `--ping`                            | Test database connectivity and exit                             |
 | `--profile`                         | Show per-statement timing summary after execution               |
 | `--progress`                        | Show a progress bar for long-running IMPORT operations          |
@@ -138,6 +140,61 @@ Run `execsql --help` for the full option list, or `execsql -m` to list all metac
 - Display query results in a GUI dialog; optionally prompt the user to select a row, enter a value, or submit a form.
 - Write status messages or tabular output to the console or a file during execution.
 - Automatically log each run, recording databases used, scripts executed, and user responses.
+- Extend with custom metacommands, exporters, and importers via the plugin system.
+
+# Library API
+
+execsql can be used as a Python library for programmatic script execution:
+
+```python
+from execsql import run
+
+# Execute a script file
+result = run(script="pipeline.sql", dsn="postgresql://user:pass@host/db")
+
+# Execute inline SQL
+result = run(
+    sql="CREATE TABLE t (id INT);\nINSERT INTO t VALUES (1);",
+    dsn="sqlite:///my.db",
+    new_db=True,
+)
+
+# With substitution variables
+result = run(
+    script="etl.sql",
+    dsn="sqlite:///data.db",
+    variables={"SCHEMA": "public", "DATE": "2026-01-01"},
+)
+
+# Check results
+print(result.success)       # True
+print(result.commands_run)  # 2
+print(result.elapsed)       # 0.003 (seconds)
+print(result.variables)     # {"SCHEMA": "public", ...}
+```
+
+Error handling:
+
+```python
+result = run(sql="SELECT * FROM nonexistent;", dsn="sqlite:///:memory:")
+if not result.success:
+    for err in result.errors:
+        print(f"{err.source}:{err.line}: {err.message}")
+
+# Or raise on failure
+result.raise_on_error()  # raises ExecSqlError
+```
+
+Use a pre-existing database connection instead of a DSN:
+
+```python
+from execsql.db.factory import db_SQLite
+conn = db_SQLite("my.db", new_db=True)
+result = run(sql="SELECT 1;", connection=conn)
+# run() does NOT close this connection — you manage its lifecycle
+```
+
+Each call to `run()` uses an isolated `RuntimeContext`, so multiple calls do not share state.
 
 # An Illustration
 

{execsql2-2.15.11 → execsql2-2.16.0}/docs/about/divergence.md

@@ -29,6 +29,8 @@ ______________________________________________________________________
 | `--profile-limit N`             | Number of top statements to display in the `--profile` summary (default: 20). Remaining statements are counted and noted in the output footer.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | `--ping`                        | Test database connectivity and exit. Connects using the supplied connection parameters, queries for the server version, and prints a one-line success message (exit 0) or the error (exit 1). No script file argument is required.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
 | `--lint`                        | Parse the script and perform static analysis without connecting to a database. Reports unmatched IF/ENDIF, LOOP/END LOOP, and BEGIN BATCH/END BATCH blocks (errors); potentially undefined `!!$VAR!!` references (warnings); missing INCLUDE file targets (warnings); and unknown `EXECUTE SCRIPT` targets (warnings). Variable analysis uses two passes so definition order does not matter. The linter descends into named script blocks reached via `EXECUTE SCRIPT` / `EXEC SCRIPT` / `RUN SCRIPT`, reads `SUB_INI` INI files at lint time, recognizes `SUB_EMPTY` / `SUB_ADD` / `SUB_APPEND` / `SUBDATA` as definitions, suppresses false warnings for `$COUNTER_N`, and auto-discovers built-in system variables from the installed source. Exits 0 if no errors, 1 if errors found. |
+| `--parse-tree`                  | Parse the script into an Abstract Syntax Tree and print a visual tree showing block nesting (IF/LOOP/BATCH/SCRIPT), source line ranges, compound conditions (ANDIF/ORIF), and all metacommands. Does not connect to a database or execute anything. Useful for understanding script structure and verifying the parser handles a script correctly.                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+| `--list-plugins`                | List all discovered plugins (metacommands, exporters, importers) from Python entry points and exit. Plugins extend execsql via `execsql.metacommands`, `execsql.exporters`, and `execsql.importers` entry point groups.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
 
 ### Export Formats
 
@@ -94,10 +96,11 @@ New options in `execsql.conf`:
 
 ### Authentication
 
-| Feature                       | Description                                                                                                                                                                    |
-| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| OS keyring integration        | When the `keyring` package is installed, passwords are stored in and retrieved from the OS credential store (macOS Keychain, Windows Credential Manager, Linux SecretService). |
-| Keyring retry on auth failure | If a stored password is rejected, the stale entry is deleted, the user is re-prompted, and the new password is saved automatically.                                            |
+| Feature                        | Description                                                                                                                                                                                       |
+| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| OS keyring integration         | When the `keyring` package is installed, passwords are stored in and retrieved from the OS credential store (macOS Keychain, Windows Credential Manager, Linux SecretService).                    |
+| Keyring retry on auth failure  | If a stored password is rejected, the stale entry is deleted, the user is re-prompted, and the new password is saved automatically.                                                               |
+| Keyring auto-store in GUI mode | When a password is entered via a GUI prompt (Tkinter or Textual), it is stored to the OS keyring automatically without an explicit confirmation prompt. CLI password prompts behave the same way. |
 
 ### Logging Enhancements
 
@@ -158,6 +161,34 @@ execsql2 adds a full interactive debugging system that has no equivalent in upst
 - **ANSI color output** — the REPL uses ANSI color on TTY outputs: bold yellow for section labels, cyan for filenames and variable names, dim for separators and `=` signs, red for error messages, bold for SQL column headers, and dim italic for `NULL` values. Color is suppressed when `NO_COLOR` or `EXECSQL_NO_COLOR` environment variables are set, or when the output stream is not a TTY.
 - **Readline support** — on platforms where `readline` is available (macOS, Linux), the REPL supports arrow-key history navigation and line editing.
 
+### Library API
+
+execsql2 provides a Python library API for programmatic script execution — no CLI needed. The upstream execsql has no equivalent.
+
+```python
+from execsql import run
+
+result = run(
+    script="pipeline.sql",
+    dsn="postgresql://user:pass@host/db",
+    variables={"SCHEMA": "public"},
+)
+
+print(result.success)       # True/False
+print(result.commands_run)  # number of statements executed
+print(result.errors)        # list of ScriptError objects
+print(result.variables)     # final substitution variable state
+```
+
+**Key features:**
+
+- **DSN or connection** — pass a DSN string (`dsn="sqlite:///my.db"`) or a pre-existing `Database` object (`connection=conn`).
+- **Substitution variables** — pass a `variables` dict; keys are automatically `$`-prefixed.
+- **Error control** — `halt_on_error=True` (default) stops on the first error; `halt_on_error=False` captures errors and continues.
+- **Isolation** — each `run()` call uses an isolated `RuntimeContext`. Multiple calls do not share state.
+- **Result object** — `ScriptResult` is a frozen dataclass with `success`, `commands_run`, `elapsed`, `errors`, and `variables`.
+- **Exception convenience** — call `result.raise_on_error()` to raise `ExecSqlError` if the script failed.
+
 ______________________________________________________________________
 
 ## Changed Behavior
@@ -192,6 +223,17 @@ All 33 mutable runtime globals in `state.py` have been consolidated into a `Runt
 - **`Database` is an ABC** — `open_db()` and `exec_cmd()` are abstract methods. Subclasses that omit them raise `TypeError` at instantiation instead of at call time.
 - **Connection timeouts** — PostgreSQL and SQLite adapters accept a connection timeout parameter (default 30 seconds).
 - **DuckDB temporal types** — `TIMESTAMPTZ`, `TIMESTAMP`, `DATE`, `TIME` now map to native DuckDB types instead of `TEXT`.
+- **SQLite `DT_Long` mapping** — `DT_Long` maps to `"hugeint"` in the SQLite type table. SQLite does not have a native `HUGEINT` type; the value receives `TEXT` affinity. In practice this is harmless because SQLite's type affinity system handles large integers transparently, but the mapping name differs from upstream.
+
+### Execution Engine
+
+The legacy flat command-list engine (`_parse_script_lines` / `runscripts` / `CommandList.run_next()`) has been replaced by an AST-based execution engine. Scripts are parsed into a tree of typed nodes, then the tree is walked for execution. This change is transparent to users — all metacommands, SQL, and control flow work identically.
+
+- **Native INCLUDE handling** — The AST executor parses INCLUDE'd files with the AST parser and executes them through the tree-walking executor. Control flow structures (IF/LOOP/BATCH/SCRIPT) in included files are fully tree-driven, with correct deferred variable handling and source-span error reporting.
+- **Circular INCLUDE detection** — The executor tracks the full include chain and detects circular references (e.g. A includes B includes A), reporting the full chain in the error message. Upstream has no such detection.
+- **BREAK outside LOOP is an error** — `BREAK` outside a loop block now raises an error (exit 1) instead of being silently ignored. This catches script bugs that upstream would not report.
+- **Instance-scoped script registry** — Named SCRIPT blocks are stored on the `RuntimeContext` instance instead of a module-level dict, preventing cross-execution contamination.
+- **Script source directory resolution** — `ScriptCmd` resolves `source_dir` at construction time (when the command is parsed) rather than per-statement at execution time. This is functionally equivalent for all normal usage since the script file does not move between parse and execution.
 
 ### Error Handling
 

{execsql2-2.15.11 → execsql2-2.16.0}/docs/api/index.md

@@ -4,7 +4,27 @@ The pages in this section are auto-generated from the source docstrings and show
 
 If you want to **extend** execsql — add a new exporter format, support a new database, or add an importer for a file type — start with the Contributing guides, which give you step-by-step walkthroughs and copy-paste skeletons. The API pages here serve as the detailed reference those guides link to.
 
-For a high-level overview of how all the pieces fit together, start with the [Architecture & Design Guide](../dev/architecture.md).
+For programmatic use, see the [Library API](#library-api) section below. For a high-level overview of how all the pieces fit together, start with the [Architecture & Design Guide](../dev/architecture.md).
+
+## Library API
+
+The primary public API is `execsql.run()`:
+
+```python
+from execsql import run, ScriptResult, ScriptError, ExecSqlError
+
+result: ScriptResult = run(
+    script="pipeline.sql",       # or sql="SELECT 1;"
+    dsn="sqlite:///my.db",       # or connection=existing_db_object
+    variables={"KEY": "value"},  # optional substitution variables
+    halt_on_error=True,          # stop on first error (default)
+    new_db=False,                # create DB if missing
+)
+```
+
+See the [README](https://github.com/geocoug/execsql#library-api) for full examples.
+
+## Extension Guides
 
 | Extension type       | Guide                                                    | API reference                   |
 | -------------------- | -------------------------------------------------------- | ------------------------------- |