execsql2 2.11.1__tar.gz → 2.12.1__tar.gz

{execsql2-2.11.1 → execsql2-2.12.1}/.claude/project_context.md

@@ -67,6 +67,9 @@ src/execsql/               # active codebase — all new work goes here
   importers/
     __init__.py
     base.py / csv.py / ods.py / xls.py / feather.py
+  debug/
+    __init__.py            # debug package
+    repl.py                # x_breakpoint, _debug_repl, REPL helpers (BREAKPOINT metacommand)
   metacommands/
     __init__.py            # DISPATCH_TABLE, format/db constants, re-exports all handlers
     dispatch.py            # build_dispatch_table() — all mcl.add() regex registrations

{execsql2-2.11.1 → execsql2-2.12.1}/CHANGELOG.md

@@ -13,6 +13,40 @@ ______________________________________________________________________
 
 ______________________________________________________________________
 
+## [2.12.1] - 2026-04-02
+
+### Changed
+
+- Performance: removed dead `_compiled_patterns` dict from `SubVarSet` — eliminated 3 unused regex compilations per `add_substitution` call (~20 calls per statement in typical scripts).
+- Performance: cached `source_dir` and `source_name` on `ScriptCmd` at construction time — eliminated per-statement `Path.resolve()` filesystem calls.
+- Performance: `select_rowdict()` now uses batched `fetchmany()` instead of row-at-a-time `fetchone()`, matching `select_rowsource()` behavior for template exports.
+- Performance: removed redundant `$CURRENT_TIME` set in `set_system_vars()` — now set once per statement in `run_and_increment()`.
+- Performance: removed no-op `copy.copy()` on immutable string in `substitute_vars()`.
+
+### Fixed
+
+- Fixed cursor leak in `select_rowsource()` — generator now closes the cursor in a `finally` block when exhausted or abandoned.
+
+______________________________________________________________________
+
+## [2.12.0] - 2026-04-01
+
+### Added
+
+- Debug REPL `.where` / `.w` command — shows current script file, line number, and the upcoming statement text (truncated to 120 chars). The entry banner now includes the location (`[Breakpoint] myscript.sql:42`) and `_print_where()` is called automatically on REPL entry via `BREAKPOINT` or step mode.
+- Debug REPL `.set VAR VAL` / `.s VAR VAL` command — sets or updates a substitution variable interactively during a `BREAKPOINT` session. Prints a confirmation line (`VAR = VAL`) on success; prints an error if substitution variables are not initialised.
+- Debug REPL ANSI color output — horizontal rule separators, colored labels ("Breakpoint"/"Step" in bold yellow, filename:line in cyan, type tags in dim green), cyan variable names, dim `=` signs, red error messages, bold SQL column headers, and dim row-count and table borders. Color is auto-detected via TTY and suppressed when `NO_COLOR` or `EXECSQL_NO_COLOR` environment variables are set. Falls back to plain text in non-interactive contexts (CI, piped output). Help text is also colorized with cyan command names and consistent column alignment.
+- Debug REPL shortcut aliases — `.h` for `.help`, `.v` for `.vars`, `.v all` for `.vars all`.
+- Debug REPL step mode banner — when the REPL is re-entered via `.next` / step mode, the entry banner now shows "Step" instead of "Breakpoint" to make it clear the pause is from stepping rather than an explicit `BREAKPOINT` metacommand.
+- `--profile-limit N` CLI option — controls how many top statements appear in the `--profile` timing summary (default: 20). The "not shown" footer message now includes the active limit for clarity.
+- Test coverage raised from 86% to 91% — 274 new tests across `metacommands/io_import.py`, `metacommands/io_export.py`, `metacommands/control.py`, `metacommands/data.py`, `importers/csv.py`, and `gui/console.py`. Coverage floor raised from 85% to 90% in `pyproject.toml`.
+
+### Changed
+
+- `execsql.debug.repl` is now a dedicated package (`src/execsql/debug/repl.py`); previously the REPL lived at `execsql.metacommands.debug_repl`. Internal import paths have been updated throughout. No public API change.
+
+______________________________________________________________________
+
 ## [2.11.1] - 2026-04-01
 
 ### Fixed

{execsql2-2.11.1 → execsql2-2.12.1}/CLAUDE.md

@@ -37,13 +37,13 @@ A multi-agent system where specialized agents collaborate to improve, extend, de
 1. **Research** — Oracle investigates codebase, finds relevant code paths and impact
 1. **Plan** — DBA synthesizes research into implementation approach, aligns with human
 1. **Implement** — Patcher writes code, Oracle advises on architecture
-1. **Test** — QA writes/runs tests, verifies coverage stays above 80%
+1. **Test** — QA writes/runs tests, verifies coverage stays above 90%
 1. **Document** — Scribe updates docs, Herald updates changelog
 1. **Review** — Inspector does final code review before human merge
 
 ## Constraints
 
-- Coverage floor (80%) must be maintained — QA blocks any change that drops it
+- Coverage floor (90%) must be maintained — QA blocks any change that drops it
 - Backwards compatibility with upstream execsql v1.130.1 unless explicitly approved
 - No destructive git operations without human approval
 - Agents should always read `.claude/project_context.md` before starting work

{execsql2-2.11.1 → execsql2-2.12.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: execsql2
-Version: 2.11.1
+Version: 2.12.1
 Summary: Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables.
 Project-URL: Repository, https://github.com/geocoug/execsql
 Project-URL: Issues, https://github.com/geocoug/execsql/issues

{execsql2-2.11.1 → execsql2-2.12.1}/docs/about/divergence.md

@@ -24,7 +24,8 @@ ______________________________________________________________________
 | `--gui-framework`               | Select GUI backend: `tkinter` (default) or `textual` (terminal UI).                                                                                                                                                                                                                                                                     |
 | `--debug`                       | Start in step-through debug mode. The debug REPL pauses before each statement, as if `BREAKPOINT` were at the top with `.next` always active.                                                                                                                                                                                           |
 | `--dry-run`                     | Parse the script and print the full command list without connecting to a database or executing anything. Substitution variables already populated at parse time (env vars, `--assign-arg` values, built-in start-time vars) are expanded in the output; execution-time variables (`$DB_NAME`, `$CURRENT_TIME`, etc.) remain unexpanded. |
-| `--profile`                     | Record wall-clock time for each SQL and metacommand statement. After the script completes, print a summary table sorted by elapsed time (descending), showing time, percentage of total, source location, command type, and a command preview. Top 20 slowest statements are shown.                                                     |
+| `--profile`                     | Record wall-clock time for each SQL and metacommand statement. After the script completes, print a summary table sorted by elapsed time (descending), showing time, percentage of total, source location, command type, and a command preview.                                                                                          |
+| `--profile-limit N`             | Number of top statements to display in the `--profile` summary (default: 20). Remaining statements are counted and noted in the output footer.                                                                                                                                                                                          |
 | `--ping`                        | Test database connectivity and exit. Connects using the supplied connection parameters, queries for the server version, and prints a one-line success message (exit 0) or the error (exit 1). No script file argument is required.                                                                                                      |
 | `--lint`                        | Parse the script and perform static analysis without connecting to a database. Reports unmatched IF/ENDIF, LOOP/END LOOP, and BEGIN BATCH/END BATCH blocks (errors), potentially undefined `!!$VAR!!` references (warnings), and missing INCLUDE file targets (warnings). Exits 0 if no errors, 1 if errors found.                      |
 
@@ -40,12 +41,12 @@ ______________________________________________________________________
 
 ### Metacommands
 
-| Metacommand            | Description                                                                                                                                                                    |
-| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| `ASSERT`               | Evaluate a condition and raise an error (halting the script) if it is false. Supports all IF conditions. Optional quoted failure message. Skipped in false IF blocks.          |
-| `BREAKPOINT`           | Pause script execution and drop into an interactive debug REPL. Inspect variables, run ad-hoc SQL, and step through the script. Silently skipped in non-TTY (CI) environments. |
-| `CONFIG SHOW_PROGRESS` | Enable the Rich progress bar for IMPORT operations at runtime.                                                                                                                 |
-| `CONFIG LOG_SQL`       | Enable SQL query audit logging — writes executed SQL to the log file.                                                                                                          |
+| Metacommand            | Description                                                                                                                                                           |
+| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `ASSERT`               | Evaluate a condition and raise an error (halting the script) if it is false. Supports all IF conditions. Optional quoted failure message. Skipped in false IF blocks. |
+| `BREAKPOINT`           | Pause script execution and drop into an interactive debug REPL. See [Debugging](#debugging) below for full details.                                                   |
+| `CONFIG SHOW_PROGRESS` | Enable the Rich progress bar for IMPORT operations at runtime.                                                                                                        |
+| `CONFIG LOG_SQL`       | Enable SQL query audit logging — writes executed SQL to the log file.                                                                                                 |
 
 ### Conditional Tests
 
@@ -106,6 +107,46 @@ New options in `execsql.conf`:
 | `py.typed` marker           | PEP 561 marker enabling downstream static type checking.                                                        |
 | Structured keyword registry | `--dump-keywords` introspects the dispatch table and outputs JSON used by the grammar generator and test suite. |
 
+### Debugging { #debugging }
+
+execsql2 adds a full interactive debugging system that has no equivalent in upstream execsql.
+
+**`BREAKPOINT` metacommand** — insert `-- !x! BREAKPOINT` anywhere in a script to pause execution and drop into a debug REPL. The REPL provides a `execsql debug>` prompt where you can inspect state and interact with the database before resuming.
+
+**`--debug` CLI flag** — start the script in step-through mode, pausing before every statement as if `BREAKPOINT` were inserted at the top with `.next` always active.
+
+**REPL commands** (all dot-prefixed to avoid collisions with variable names and SQL):
+
+| Command                | Description                                                               |
+| ---------------------- | ------------------------------------------------------------------------- |
+| `.continue` / `.c`     | Resume normal script execution                                            |
+| `.abort` / `.q`        | Halt the script with exit status 1                                        |
+| `.vars` / `.v`         | List all user, system, local, and counter variables (grouped by type)     |
+| `.vars all` / `.v all` | Include environment variables (`&`) in the listing                        |
+| `.next` / `.n`         | Execute the next statement, then pause again (step mode)                  |
+| `.where` / `.w`        | Show the current script file, line number, and upcoming statement text    |
+| `.stack`               | Show the command-list stack (script name, cursor position, nesting depth) |
+| `.set VAR VAL` / `.s`  | Set or update a substitution variable; prints confirmation on success     |
+| `.help` / `.h`         | Show available commands                                                   |
+
+**Non-prefixed input** is interpreted as either a variable lookup or ad-hoc SQL:
+
+| Input                          | Behavior                                                                         |
+| ------------------------------ | -------------------------------------------------------------------------------- |
+| `logfile`                      | Print the value of the `logfile` substitution variable                           |
+| `$ARG_1`                       | Print the value of a system/built-in variable                                    |
+| `SELECT count(*) FROM orders;` | Execute SQL against the current database and pretty-print the results in a table |
+
+**Key behaviors:**
+
+- **Non-interactive safety** — `BREAKPOINT` is silently skipped when `sys.stdin` is not a TTY (CI pipelines, piped input, cron). Scripts never hang in automation.
+- **Ad-hoc SQL** — any input ending with `;` is executed against the current database connection using the same cursor and transaction state as the script. Queries return a formatted table; DML statements (INSERT, UPDATE, DELETE) execute and commit/rollback according to the current autocommit and batch settings.
+- **Variable inspection** — bare names look up user-defined variables (e.g., `logfile`); sigil-prefixed names look up system (`$`), environment (`&`), local (`~`), or counter (`@`) variables. If a `$`-prefixed name isn't found, the REPL strips the sigil and retries (since `SUB` stores keys without a prefix).
+- **Step mode** — `.next` executes exactly one statement then re-enters the REPL. When stepping, the entry banner shows "Step" instead of "Breakpoint" to distinguish stepping from an explicit `BREAKPOINT`. Combined with `.where`, `.vars`, and SQL queries, this allows line-by-line script debugging with full state visibility.
+- **Location display** — on entry to the REPL (via `BREAKPOINT` or step mode) the banner shows a horizontal rule with the label ("Breakpoint" or "Step"), the current filename and line number, and the upcoming statement. Use `.where` (or `.w`) at any time to re-display this information.
+- **ANSI color output** — the REPL uses ANSI color on TTY outputs: bold yellow for section labels, cyan for filenames and variable names, dim for separators and `=` signs, red for error messages, bold for SQL column headers, and dim italic for `NULL` values. Color is suppressed when `NO_COLOR` or `EXECSQL_NO_COLOR` environment variables are set, or when the output stream is not a TTY.
+- **Readline support** — on platforms where `readline` is available (macOS, Linux), the REPL supports arrow-key history navigation and line editing.
+
 ______________________________________________________________________
 
 ## Changed Behavior

{execsql2-2.11.1 → execsql2-2.12.1}/docs/guides/debugging.md

@@ -38,6 +38,47 @@ DEBUG WRITE <script_name> [[APPEND] TO <filename>]
 
 This is an alias for the [WRITE SCRIPT](../reference/metacommands.md#write_script) metacommand.
 
+# Interactive Debug REPL (BREAKPOINT)
+
+Insert `-- !x! BREAKPOINT` anywhere in a script to pause execution and drop into the interactive debug REPL:
+
+```sql
+-- !x! BREAKPOINT
+SELECT * FROM orders WHERE status = 'pending';
+```
+
+The REPL prints the current file name, line number, and the upcoming statement when it opens:
+
+```
+[Breakpoint] myscript.sql:42 — Script paused. Type '.help' for commands, '.c' to resume.
+  myscript.sql:42  (sql)
+  → SELECT * FROM orders WHERE status = 'pending';
+execsql debug>
+```
+
+**Available commands:**
+
+| Command        | Shortcut | Description                                                               |
+| -------------- | -------- | ------------------------------------------------------------------------- |
+| `.continue`    | `.c`     | Resume normal script execution                                            |
+| `.abort`       | `.q`     | Halt the script with exit status 1                                        |
+| `.vars`        | `.v`     | List user, system, local, and counter substitution variables              |
+| `.vars all`    | `.v all` | Include environment variables (`&`) in the listing                        |
+| `.next`        | `.n`     | Execute the next statement, then pause again (step mode)                  |
+| `.where`       | `.w`     | Re-display the current script location and upcoming statement             |
+| `.stack`       |          | Show the command-list stack (script name, cursor position, nesting depth) |
+| `.set VAR VAL` | `.s`     | Set or update a substitution variable                                     |
+| `.help`        | `.h`     | Show available commands                                                   |
+
+Anything not starting with `.` is treated as a variable lookup or SQL:
+
+- A bare name (e.g. `logfile`) prints the value of that substitution variable.
+- Any input ending with `;` is executed as SQL against the current database (expects columns returned, e.g. SELECT).
+
+The `--debug` CLI flag starts execution in step mode, pausing before every statement.
+
+In non-interactive environments (CI, piped input) `BREAKPOINT` is silently skipped so automated pipelines are never blocked.
+
 The ON ERROR_HALT metacommands allow custom reporting (or cleanup) actions to be taken when errors occur.
 
 Setting the configuration setting [write_warnings](../reference/configuration.md#write_warnings) to "Yes" can also assist with debugging by displaying conditions that may result from errors in the script.

{execsql2-2.11.1 → execsql2-2.12.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "execsql2"
-version = "2.11.1"
+version = "2.12.1"
 description = "Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables."
 readme = { file = "README.md", content-type = "text/markdown" }
 license = { file = "LICENSE.txt" }
@@ -158,7 +158,7 @@ skip-magic-trailing-comma = false
 line-ending = "auto"
 
 [tool.bumpversion]
-current_version = "2.11.1"
+current_version = "2.12.1"
 commit = true
 commit_args = "--no-verify"
 tag = true
@@ -182,7 +182,7 @@ addopts = [
     "--cov=execsql",
     "--cov-branch",
     "--cov-report=xml",
-    "--cov-fail-under=85",
+    "--cov-fail-under=90",
     "--strict-markers",
     "--strict-config",
     "--color=yes",

{execsql2-2.11.1 → execsql2-2.12.1}/src/execsql/cli/__init__.py

@@ -278,6 +278,11 @@ def main(
         "--profile",
         help="Record per-statement execution times and print a timing summary after the script completes.",
     ),
+    profile_limit: int = typer.Option(
+        20,
+        "--profile-limit",
+        help="Number of top statements to show in the --profile timing summary (default: 20).",
+    ),
     debug: bool = typer.Option(
         False,
         "--debug",
@@ -449,6 +454,7 @@ def main(
         output_dir=output_dir,
         progress=progress,
         profile=profile,
+        profile_limit=profile_limit,
         ping=ping,
         lint=lint,
         debug=debug,

{execsql2-2.11.1 → execsql2-2.12.1}/src/execsql/cli/run.py

@@ -71,12 +71,14 @@ def _print_dry_run(cmdlist: object) -> None:
 # ---------------------------------------------------------------------------
 
 
-def _print_profile(profile_data: list[tuple]) -> None:
+def _print_profile(profile_data: list[tuple], limit: int = 20) -> None:
     """Print a per-statement timing summary to stdout.
 
     Args:
         profile_data: List of ``(source, line_no, command_type, elapsed_secs,
             command_text_preview)`` tuples collected during execution.
+        limit: Maximum number of top statements to display (default: 20).
+            All statements are included in totals regardless of this value.
     """
     if not profile_data:
         _console.print("[dim]Profile: no statements recorded.[/dim]")
@@ -85,9 +87,9 @@ def _print_profile(profile_data: list[tuple]) -> None:
     total_secs = sum(row[3] for row in profile_data)
     n = len(profile_data)
 
-    # Sort descending by elapsed time; show top 20 (or all if <= 20).
+    # Sort descending by elapsed time; show top `limit` (or all if <= limit).
     sorted_data = sorted(profile_data, key=lambda r: r[3], reverse=True)
-    display = sorted_data[:20]
+    display = sorted_data[:limit]
 
     _console.print()
     _console.print(f"[bold cyan]Profile:[/bold cyan] {n} statement{'s' if n != 1 else ''} in {total_secs:.3f}s")
@@ -115,10 +117,10 @@ def _print_profile(profile_data: list[tuple]) -> None:
             f"{preview_short}",
         )
 
-    if len(sorted_data) > 20:
-        omitted = len(sorted_data) - 20
+    if len(sorted_data) > limit:
+        omitted = len(sorted_data) - limit
         _console.print(
-            f"[dim]  ... {omitted} more statement{'s' if omitted != 1 else ''} not shown (top 20 by time)[/dim]",
+            f"[dim]  ... {omitted} more statement{'s' if omitted != 1 else ''} not shown (top {limit} by time)[/dim]",
         )
 
     _console.print()
@@ -213,6 +215,7 @@ def _run(
     output_dir: str | None = None,
     progress: bool = False,
     profile: bool = False,
+    profile_limit: int = 20,
     ping: bool = False,
     lint: bool = False,
     debug: bool = False,
@@ -552,7 +555,7 @@ def _run(
     if debug:
         _state.step_mode = True
 
-    _execute_script_direct(conf, profile=profile)
+    _execute_script_direct(conf, profile=profile, profile_limit=profile_limit)
 
 
 # ---------------------------------------------------------------------------
@@ -611,7 +614,7 @@ def _execute_script_textual_console(conf: ConfigData) -> None:
     _state.exec_log.log_exit_end()
 
 
-def _execute_script_direct(conf: ConfigData, *, profile: bool = False) -> None:
+def _execute_script_direct(conf: ConfigData, *, profile: bool = False, profile_limit: int = 20) -> None:
     """Run runscripts() in the current (main) thread — used when Textual is not active.
 
     Args:
@@ -619,6 +622,8 @@ def _execute_script_direct(conf: ConfigData, *, profile: bool = False) -> None:
         profile: When ``True``, print a per-statement timing summary after the
             script completes.  Timing data must already have been activated on
             ``_state.profile_data`` before this function is called.
+        profile_limit: Maximum number of top statements to display in the
+            profile summary (default: 20).
     """
     import execsql.state as _state
     import execsql.utils.gui as _gui
@@ -645,7 +650,7 @@ def _execute_script_direct(conf: ConfigData, *, profile: bool = False) -> None:
                 gui_console_off()
         _state.exec_log.log_status_info(f"{_state.cmds_run} commands run")
         if profile and _state.profile_data is not None:
-            _print_profile(_state.profile_data)
+            _print_profile(_state.profile_data, limit=profile_limit)
         sys.exit(exc.code)
     except ConfigError:
         raise
@@ -673,7 +678,7 @@ def _execute_script_direct(conf: ConfigData, *, profile: bool = False) -> None:
             gui_console_off()
     _state.exec_log.log_status_info(f"{_state.cmds_run} commands run")
     if profile and _state.profile_data is not None:
-        _print_profile(_state.profile_data)
+        _print_profile(_state.profile_data, limit=profile_limit)
     _state.exec_log.log_exit_end()
 
 

{execsql2-2.11.1 → execsql2-2.12.1}/src/execsql/db/base.py

@@ -234,18 +234,22 @@ class Database(ABC):
             pass  # Non-critical: some drivers lack rowcount support.
 
         def decode_row() -> Generator:
-            while True:
-                rows = curs.fetchmany()
-                if not rows:
-                    break
-                else:
-                    for row in rows:
-                        if self.encoding:
-                            yield [
-                                c.decode(self.encoding, "backslashreplace") if isinstance(c, bytes) else c for c in row
-                            ]
-                        else:
-                            yield row
+            try:
+                while True:
+                    rows = curs.fetchmany()
+                    if not rows:
+                        break
+                    else:
+                        for row in rows:
+                            if self.encoding:
+                                yield [
+                                    c.decode(self.encoding, "backslashreplace") if isinstance(c, bytes) else c
+                                    for c in row
+                                ]
+                            else:
+                                yield row
+            finally:
+                curs.close()
 
         return [d[0] for d in curs.description], decode_row()
 
@@ -253,6 +257,10 @@ class Database(ABC):
         """Execute *sql* and return ``(column_names, row_iterator)`` where each row is a ``dict``."""
         # Return an iterable that yields dictionaries of row data
         curs = self.cursor()
+        try:
+            curs.arraysize = _state.conf.export_row_buffer
+        except Exception:
+            pass  # Non-critical: not all drivers support arraysize.
         try:
             curs.execute(sql)
         except Exception:
@@ -264,18 +272,24 @@ class Database(ABC):
             pass  # Non-critical: some drivers lack rowcount support.
         hdrs = [d[0] for d in curs.description]
 
-        def dict_row() -> dict | None:
-            row = curs.fetchone()
-            if row:
-                if self.encoding:
-                    r = [c.decode(self.encoding, "backslashreplace") if isinstance(c, bytes) else c for c in row]
-                else:
-                    r = row
-                return dict(zip(hdrs, r))
-            else:
-                return None
+        def dict_rows() -> Generator:
+            try:
+                while True:
+                    rows = curs.fetchmany()
+                    if not rows:
+                        break
+                    for row in rows:
+                        if self.encoding:
+                            r = [
+                                c.decode(self.encoding, "backslashreplace") if isinstance(c, bytes) else c for c in row
+                            ]
+                        else:
+                            r = row
+                        yield dict(zip(hdrs, r))
+            finally:
+                curs.close()
 
-        return hdrs, iter(dict_row, None)
+        return hdrs, dict_rows()
 
     def schema_exists(self, schema_name: str) -> bool:
         """Return ``True`` if *schema_name* exists in this database."""

execsql2-2.12.1/src/execsql/debug/__init__.py

@@ -0,0 +1,6 @@
+"""Debug utilities for execsql.
+
+This package contains interactive debugging tools for execsql scripts:
+
+- :mod:`execsql.debug.repl` — the ``BREAKPOINT`` metacommand REPL.
+"""