execsql2 2.14.1__tar.gz → 2.15.0__tar.gz

{execsql2-2.14.1 → execsql2-2.15.0}/CHANGELOG.md

@@ -13,8 +13,28 @@ ______________________________________________________________________
 
 ______________________________________________________________________
 
+## [2.15.0] - 2026-04-09
+
+### Added
+
+- `PG_UPSERT` metacommand: new `EXPORT_FAILURES <dir>`, `EXPORT_FORMAT csv|json|xlsx`, and `EXPORT_MAX_ROWS <n>` keywords that write a "fix sheet" of failing QA rows — one row per unique violating staging row with a consolidated `_issues` column — to CSV, JSON, or XLSX. Works in all three modes (full pipeline, QA-only, schema check) and runs even when QA fails. New `$PG_UPSERT_EXPORT_PATH` substitution variable holds the directory written. A user-visible message reporting the export directory and format is emitted to both the console and the execsql log after every export.
+
+### Changed
+
+- `[upsert]` extra now requires `pg-upsert>=1.21.0` (up from `>=1.20.0`) for the fix-sheet export feature.
+
+### Fixed
+
+- `PROMPT MESSAGE ... CREDENTIALS <user_var> <pw_var>` no longer crashes in console-fallback mode with `TypeError: get_password() missing 2 required positional arguments: 'database_name' and 'user_name'`. The fallback now uses `getpass.getpass()` to read the password, matching the intent (keyring-aware `auth.get_password()` is for CONNECT, not for bare credential prompts).
+
+______________________________________________________________________
+
 ## [2.14.1] - 2026-04-07
 
+### Fixed
+
+- Fix Windows CI: use `zf.namelist()[0]` instead of path for zip entry lookup.
+
 ______________________________________________________________________
 
 ## [2.14.0] - 2026-04-07

{execsql2-2.14.1 → execsql2-2.15.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: execsql2
-Version: 2.14.1
+Version: 2.15.0
 Summary: Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables.
 Project-URL: Homepage, https://execsql2.readthedocs.io
 Project-URL: Repository, https://github.com/geocoug/execsql
@@ -51,7 +51,7 @@ Requires-Dist: keyring; extra == 'all'
 Requires-Dist: odfpy; extra == 'all'
 Requires-Dist: openpyxl; extra == 'all'
 Requires-Dist: oracledb; extra == 'all'
-Requires-Dist: pg-upsert>=1.20.0; extra == 'all'
+Requires-Dist: pg-upsert>=1.21.0; extra == 'all'
 Requires-Dist: polars; extra == 'all'
 Requires-Dist: psycopg2-binary; extra == 'all'
 Requires-Dist: pymysql; extra == 'all'
@@ -109,7 +109,7 @@ Requires-Dist: oracledb; extra == 'oracle'
 Provides-Extra: postgres
 Requires-Dist: psycopg2-binary; extra == 'postgres'
 Provides-Extra: upsert
-Requires-Dist: pg-upsert>=1.20.0; extra == 'upsert'
+Requires-Dist: pg-upsert>=1.21.0; extra == 'upsert'
 Description-Content-Type: text/markdown
 
 > [!NOTE]

{execsql2-2.14.1 → execsql2-2.15.0}/docs/about/divergence.md

@@ -41,14 +41,14 @@ ______________________________________________________________________
 
 ### Metacommands
 
-| Metacommand            | Description                                                                                                                                                                                                                      |
-| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `ASSERT`               | Evaluate a condition and raise an error (halting the script) if it is false. Supports all IF conditions. Optional quoted failure message. Skipped in false IF blocks.                                                            |
-| `BREAKPOINT`           | Pause script execution and drop into an interactive debug REPL. See [Debugging](#debugging) below for full details.                                                                                                              |
-| `CONFIG SHOW_PROGRESS` | Enable the Rich progress bar for IMPORT operations at runtime.                                                                                                                                                                   |
-| `CONFIG LOG_SQL`       | Enable SQL query audit logging — writes executed SQL to the log file.                                                                                                                                                            |
-| `PG_UPSERT`            | QA-checked, FK-dependency-ordered upserts from staging to base schema on PostgreSQL. Integrates [pg-upsert](https://pg-upsert.readthedocs.io/) as an optional dependency. Three modes: full pipeline, QA-only, and schema check. |
-| `IMPORT … FROM JSON`   | Import a JSON file (array of objects or NDJSON) into a database table. Nested objects are flattened with dot-separated column names; arrays are stored as JSON strings.                                                          |
+| Metacommand            | Description                                                                                                                                                                                                                                                                                                                                                                                             |
+| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `ASSERT`               | Evaluate a condition and raise an error (halting the script) if it is false. Supports all IF conditions. Optional quoted failure message. Skipped in false IF blocks.                                                                                                                                                                                                                                   |
+| `BREAKPOINT`           | Pause script execution and drop into an interactive debug REPL. See [Debugging](#debugging) below for full details.                                                                                                                                                                                                                                                                                     |
+| `CONFIG SHOW_PROGRESS` | Enable the Rich progress bar for IMPORT operations at runtime.                                                                                                                                                                                                                                                                                                                                          |
+| `CONFIG LOG_SQL`       | Enable SQL query audit logging — writes executed SQL to the log file.                                                                                                                                                                                                                                                                                                                                   |
+| `PG_UPSERT`            | QA-checked, FK-dependency-ordered upserts from staging to base schema on PostgreSQL. Integrates [pg-upsert](https://pg-upsert.readthedocs.io/) as an optional dependency. Three modes: full pipeline, QA-only, and schema check. Supports `EXPORT_FAILURES`, `EXPORT_FORMAT`, and `EXPORT_MAX_ROWS` keywords to write a "fix sheet" of failing QA rows to CSV/JSON/XLSX (requires `pg-upsert>=1.21.0`). |
+| `IMPORT … FROM JSON`   | Import a JSON file (array of objects or NDJSON) into a database table. Nested objects are flattened with dot-separated column names; arrays are stored as JSON strings.                                                                                                                                                                                                                                 |
 
 ### Conditional Tests
 

{execsql2-2.14.1 → execsql2-2.15.0}/docs/reference/metacommands.md

@@ -2127,6 +2127,9 @@ Keywords can appear in any order after the table list.
 | `COMPACT` | Use compact grid format for QA summary instead of detailed per-table panels. |
 | `LOGFILE <path>` | Append pg-upsert's plain-text log output to the given file. Supports quoted paths for spaces: `LOGFILE "path/to/log.txt"`. |
 | `CLEANUP` | Drop all `ups_*` temporary tables and views after execution. Without it, temp objects persist for inspection (default). |
+| `EXPORT_FAILURES <dir>` | Write a "fix sheet" of failing QA rows into `<dir>` (directory is created if missing). One row per unique violating staging row, with an `_issues` column summarizing every problem on that row. Supports quoted paths for spaces. Export runs even when QA fails — that is the whole point. A confirmation message (`PG_UPSERT: exported QA failures to <dir> (<format>)`) is written to the terminal, the execsql log, and the `LOGFILE` target if one is given. |
+| `EXPORT_FORMAT csv\|json\|xlsx` | Fix sheet format when `EXPORT_FAILURES` is given. `csv` (default) writes one file per table; `json` writes a single nested file; `xlsx` writes a single workbook with one sheet per table (requires `openpyxl`). |
+| `EXPORT_MAX_ROWS <n>` | Maximum rows to capture per check per table for the fix sheet. Default `1000`. Only meaningful with `EXPORT_FAILURES`. |
 
 ### Substitution variables
 
@@ -2150,6 +2153,7 @@ Set after every `PG_UPSERT` execution:
 | `$PG_UPSERT_TABLE_QA_PASSED` | TRUE/FALSE | QA result for the current table (updated per table) |
 | `$PG_UPSERT_TABLE_ROWS_UPDATED` | integer | Rows updated for the current table (updated per table) |
 | `$PG_UPSERT_TABLE_ROWS_INSERTED` | integer | Rows inserted for the current table (updated per table) |
+| `$PG_UPSERT_EXPORT_PATH` | string | Directory path that the QA fix sheet was written to, or empty if `EXPORT_FAILURES` was not given or nothing was exported. |
 
 !!! note "Using `$PG_UPSERT_RESULT_JSON` with WRITE"
     The JSON value is stored as compact single-line JSON. Because it contains double quotes (`"`), square brackets (`[]`), and apostrophes may appear in data, use tilde or backtick delimiters with WRITE:
@@ -2211,6 +2215,13 @@ For the full list of temporary objects and their schemas, see the [pg-upsert Tem
 -- Write the full JSON result using tilde delimiters (JSON contains " and [])
 -- !x! PG_UPSERT FROM staging TO public TABLES books COMMIT
 -- !x! WRITE ~!!$PG_UPSERT_RESULT_JSON!!~
+
+-- Export failing QA rows to an Excel fix sheet (one sheet per table)
+-- !x! PG_UPSERT QA FROM staging TO public TABLES books, authors EXPORT_FAILURES "qa_failures/" EXPORT_FORMAT xlsx
+
+-- Full pipeline with a CSV fix sheet cap of 50 rows per check per table
+-- !x! PG_UPSERT FROM staging TO public TABLES books, authors EXPORT_FAILURES /tmp/fix EXPORT_MAX_ROWS 50 COMMIT
+-- !x! WRITE "Fix sheet: !!$PG_UPSERT_EXPORT_PATH!!"
 ```
 
 

{execsql2-2.14.1 → execsql2-2.15.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "execsql2"
-version = "2.14.1"
+version = "2.15.0"
 description = "Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables."
 readme = { file = "README.md", content-type = "text/markdown" }
 license = { file = "LICENSE.txt" }
@@ -58,7 +58,7 @@ odbc = ["pyodbc"]
 # Feature bundles
 formats = ["odfpy", "xlrd", "openpyxl", "Jinja2", "polars", "tables", "PyYAML"]
 auth = ["keyring"]
-upsert = ["pg-upsert>=1.20.0"]
+upsert = ["pg-upsert>=1.21.0"]
 # Convenience groups
 all-db = [
     "psycopg2-binary",
@@ -161,7 +161,7 @@ skip-magic-trailing-comma = false
 line-ending = "auto"
 
 [tool.bumpversion]
-current_version = "2.14.1"
+current_version = "2.15.0"
 commit = true
 commit_args = "--no-verify"
 tag = true

{execsql2-2.14.1 → execsql2-2.15.0}/src/execsql/metacommands/upsert.py

@@ -26,11 +26,11 @@ from execsql.utils.errors import exception_desc
 
 _KW_METHOD = re.compile(r"\bMETHOD\s+(upsert|update|insert)\b", re.IGNORECASE)
 _KW_EXCLUDE = re.compile(
-    r"\bEXCLUDE\s+([\w\s,]+?)(?=\s+(?:METHOD|COMMIT|INTERACTIVE|COMPACT|EXCLUDE_NULL|LOGFILE|CLEANUP)\b|\s*$)",
+    r"\bEXCLUDE\s+([\w\s,]+?)(?=\s+(?:METHOD|COMMIT|INTERACTIVE|COMPACT|EXCLUDE_NULL|LOGFILE|CLEANUP|EXPORT_FAILURES|EXPORT_FORMAT|EXPORT_MAX_ROWS)\b|\s*$)",
     re.IGNORECASE,
 )
 _KW_EXCLUDE_NULL = re.compile(
-    r"\bEXCLUDE_NULL\s+([\w\s,]+?)(?=\s+(?:METHOD|COMMIT|INTERACTIVE|COMPACT|EXCLUDE|LOGFILE|CLEANUP)\b|\s*$)",
+    r"\bEXCLUDE_NULL\s+([\w\s,]+?)(?=\s+(?:METHOD|COMMIT|INTERACTIVE|COMPACT|EXCLUDE|LOGFILE|CLEANUP|EXPORT_FAILURES|EXPORT_FORMAT|EXPORT_MAX_ROWS)\b|\s*$)",
     re.IGNORECASE,
 )
 _KW_COMMIT = re.compile(r"\bCOMMIT\b", re.IGNORECASE)
@@ -38,13 +38,21 @@ _KW_INTERACTIVE = re.compile(r"\bINTERACTIVE\b", re.IGNORECASE)
 _KW_COMPACT = re.compile(r"\bCOMPACT\b", re.IGNORECASE)
 _KW_CLEANUP = re.compile(r"\bCLEANUP\b", re.IGNORECASE)
 _KW_LOGFILE = re.compile(r"""\bLOGFILE\s+(?:"([^"]+)"|'([^']+)'|(\S+))""", re.IGNORECASE)
+_KW_EXPORT_FAILURES = re.compile(
+    r"""\bEXPORT_FAILURES\s+(?:"([^"]+)"|'([^']+)'|(\S+))""",
+    re.IGNORECASE,
+)
+_KW_EXPORT_FORMAT = re.compile(r"\bEXPORT_FORMAT\s+(\S+)", re.IGNORECASE)
+_KW_EXPORT_MAX_ROWS = re.compile(r"\bEXPORT_MAX_ROWS\s+(\S+)", re.IGNORECASE)
 
 # All recognized keywords — used to split table names from options.
 _ALL_KEYWORDS = re.compile(
-    r"\b(?:METHOD|COMMIT|INTERACTIVE|COMPACT|EXCLUDE_NULL|EXCLUDE|LOGFILE|CLEANUP)\b",
+    r"\b(?:METHOD|COMMIT|INTERACTIVE|COMPACT|EXCLUDE_NULL|EXCLUDE|LOGFILE|CLEANUP|EXPORT_FAILURES|EXPORT_FORMAT|EXPORT_MAX_ROWS)\b",
     re.IGNORECASE,
 )
 
+_VALID_EXPORT_FORMATS = ("csv", "json", "xlsx")
+
 
 def _parse_tables_and_options(tail: str) -> dict[str, Any]:
     """Parse the trailing text after ``TABLES`` into table names and options.
@@ -90,6 +98,40 @@ def _parse_tables_and_options(tail: str) -> dict[str, Any]:
     if m:
         logfile = m.group(1) or m.group(2) or m.group(3)
 
+    export_failures: str | None = None
+    m = _KW_EXPORT_FAILURES.search(opts_part)
+    if m:
+        export_failures = m.group(1) or m.group(2) or m.group(3)
+
+    export_format = "csv"
+    m = _KW_EXPORT_FORMAT.search(opts_part)
+    if m:
+        fmt = m.group(1).lower()
+        if fmt not in _VALID_EXPORT_FORMATS:
+            raise ErrInfo(
+                "cmd",
+                other_msg=(
+                    f"PG_UPSERT: unsupported EXPORT_FORMAT {m.group(1)!r}. "
+                    f"Supported: {', '.join(_VALID_EXPORT_FORMATS)}"
+                ),
+            )
+        export_format = fmt
+
+    export_max_rows = 1000
+    m = _KW_EXPORT_MAX_ROWS.search(opts_part)
+    if m:
+        raw = m.group(1)
+        try:
+            val = int(raw)
+            if val <= 0:
+                raise ValueError
+        except ValueError as exc:
+            raise ErrInfo(
+                "cmd",
+                other_msg=(f"PG_UPSERT: EXPORT_MAX_ROWS must be a positive integer, got {raw!r}"),
+            ) from exc
+        export_max_rows = val
+
     return {
         "tables": tables,
         "method": method,
@@ -100,6 +142,9 @@ def _parse_tables_and_options(tail: str) -> dict[str, Any]:
         "exclude_null_check_cols": exclude_null,
         "logfile": logfile,
         "cleanup": bool(_KW_CLEANUP.search(opts_part)),
+        "export_failures": export_failures,
+        "export_format": export_format,
+        "export_max_rows": export_max_rows,
     }
 
 
@@ -157,6 +202,9 @@ def _set_subvars(result: Any) -> None:
     sv("$PG_UPSERT_STARTED_AT", result.started_at)
     sv("$PG_UPSERT_FINISHED_AT", result.finished_at)
     sv("$PG_UPSERT_RESULT_JSON", json.dumps(result.to_dict(), separators=(",", ":")))
+    # Default export path subvar to empty; _export_failures_if_requested
+    # will overwrite it with the actual path if an export was produced.
+    sv("$PG_UPSERT_EXPORT_PATH", "")
 
 
 def _qa_failure_msg(result: Any) -> str:
@@ -241,20 +289,26 @@ def _create_pgupsert(
     if _state.conf:
         ui_mode = _state.conf.gui_framework
 
-    ups = PgUpsert(
-        conn=db.conn,
-        staging_schema=staging_schema,
-        base_schema=base_schema,
-        tables=opts["tables"],
-        do_commit=opts["commit"],
-        interactive=opts["interactive"],
-        compact=opts["compact"],
-        upsert_method=opts["method"],
-        exclude_cols=opts["exclude_cols"],
-        exclude_null_check_cols=opts["exclude_null_check_cols"],
-        ui_mode=ui_mode,
-        callback=_make_callback(),
-    )
+    kwargs: dict[str, Any] = {
+        "conn": db.conn,
+        "staging_schema": staging_schema,
+        "base_schema": base_schema,
+        "tables": opts["tables"],
+        "do_commit": opts["commit"],
+        "interactive": opts["interactive"],
+        "compact": opts["compact"],
+        "upsert_method": opts["method"],
+        "exclude_cols": opts["exclude_cols"],
+        "exclude_null_check_cols": opts["exclude_null_check_cols"],
+        "ui_mode": ui_mode,
+        "callback": _make_callback(),
+    }
+    # Only pass fix-sheet capture args when an export was requested, so the
+    # metacommand stays compatible with any pg-upsert build that lacks them.
+    if opts.get("export_failures"):
+        kwargs["capture_detail_rows"] = True
+        kwargs["max_export_rows"] = opts.get("export_max_rows", 1000)
+    ups = PgUpsert(**kwargs)
     return ups
 
 
@@ -328,6 +382,57 @@ def _run_with_autocommit_guard(db: Any, fn: Any) -> Any:
             db.autocommit_on()
 
 
+def _export_failures_if_requested(
+    result: Any,
+    opts: dict[str, Any],
+    metacommandline: str | None,
+) -> None:
+    """Export a QA fix sheet if EXPORT_FAILURES was given in the metacommand.
+
+    Always called after ``_set_subvars(result)`` so ``$PG_UPSERT_EXPORT_PATH``
+    is initialized to empty first, then overwritten here on a successful
+    export.  Called even when QA failed — that's the whole point of the
+    fix sheet.
+    """
+    path = opts.get("export_failures")
+    if not path:
+        return
+    fmt = opts["export_format"]
+    try:
+        exported = result.export_failures(path, fmt=fmt)
+    except Exception as exc:
+        raise ErrInfo(
+            "exception",
+            exception_msg=exception_desc(),
+            other_msg=(f"PG_UPSERT failed to export failure sheet to {path}"),
+        ) from exc
+    _state.subvars.add_substitution(
+        "$PG_UPSERT_EXPORT_PATH",
+        str(exported) if exported else "",
+    )
+    if exported:
+        msg = f"PG_UPSERT: exported QA failures to {exported} ({fmt})"
+    else:
+        msg = f"PG_UPSERT: no QA failures to export (EXPORT_FAILURES {path} skipped)"
+    _state.exec_log.log_user_msg(msg)
+    try:
+        _state.output.write(msg + "\n")
+    except Exception:
+        # Output sink may be unavailable in some contexts (e.g. tests);
+        # the log message above is sufficient in that case.
+        pass
+    # Tee to the LOGFILE keyword target if one was given, matching how the
+    # pg-upsert display output is routed there via _FileWriterHandler.
+    logfile = opts.get("logfile")
+    if logfile:
+        try:
+            from execsql.utils.fileio import filewriter_write
+
+            filewriter_write(logfile, msg + "\n")
+        except Exception:
+            pass
+
+
 def _handle_pg_upsert_errors(fn: Any, metacommandline: str | None) -> Any:
     """Run *fn*, translating pg-upsert exceptions to ErrInfo."""
     from pg_upsert import UserCancelledError
@@ -381,6 +486,7 @@ def x_pg_upsert(**kwargs: Any) -> None:
         _detach_log_handlers(loggers, handlers, prev_levels)
 
     _set_subvars(result)
+    _export_failures_if_requested(result, opts, metacommandline)
     if opts.get("cleanup"):
         ups.cleanup()
 
@@ -420,6 +526,7 @@ def x_pg_upsert_qa(**kwargs: Any) -> None:
 
     result = _build_result_from_qa_errors(ups)
     _set_subvars(result)
+    _export_failures_if_requested(result, opts, metacommandline)
     if opts.get("cleanup"):
         ups.cleanup()
 
@@ -462,6 +569,7 @@ def x_pg_upsert_check(**kwargs: Any) -> None:
 
     result = _build_result_from_qa_errors(ups)
     _set_subvars(result)
+    _export_failures_if_requested(result, opts, metacommandline)
     if opts.get("cleanup"):
         ups.cleanup()
 

{execsql2-2.14.1 → execsql2-2.15.0}/src/execsql/utils/gui.py

@@ -498,10 +498,10 @@ def gui_credentials(
     cmd:
         The originating metacommand line (for logging only).
     """
+    import getpass as _getpass
     import queue as _queue
 
     import execsql.state as _state
-    from execsql.utils.auth import get_password
 
     gui_level = _state.conf.gui_level if _state.conf else 0
     if gui_level > 0 and _state.gui_manager_thread is not None and _state.gui_manager_thread.is_alive():
@@ -515,7 +515,7 @@ def gui_credentials(
         if message:
             print(message, file=sys.stderr)
         uname = input("Username: ")
-        passwd = get_password(f"Password for {uname}: ")
+        passwd = _getpass.getpass(f"Password for {uname}: ")
 
     if username:
         _state.subvars.add_substitution(username, uname)

{execsql2-2.14.1 → execsql2-2.15.0}/tests/gui/test_backends.py

@@ -808,6 +808,34 @@ class TestGuiPublicAPI:
         monkeypatch.setattr("builtins.input", lambda *a: next(inputs))
         assert get_yn("?") is False
 
+    def test_gui_credentials_console_fallback(self, monkeypatch):
+        """gui_credentials must not call auth.get_password (which needs
+        dbms/database/user kwargs). Regression for the
+        'get_password() missing 2 required positional arguments' bug.
+        """
+        from unittest.mock import MagicMock
+
+        import execsql.state as _state
+        from execsql.utils.gui import gui_credentials
+
+        # Force the fallback branch: no GUI manager thread.
+        monkeypatch.setattr(_state, "gui_manager_thread", None)
+        conf = MagicMock()
+        conf.gui_level = 0
+        monkeypatch.setattr(_state, "conf", conf)
+
+        subvars = MagicMock()
+        monkeypatch.setattr(_state, "subvars", subvars)
+
+        monkeypatch.setattr("builtins.input", lambda *a: "alice")
+        monkeypatch.setattr("getpass.getpass", lambda *a, **kw: "s3cret")
+
+        gui_credentials(message="Log in", username="$U", pwtext="$P")
+
+        calls = {c[0][0]: c[0][1] for c in subvars.add_substitution.call_args_list}
+        assert calls["$U"] == "alice"
+        assert calls["$P"] == "s3cret"
+
     def test_pause_no_countdown(self, monkeypatch):
         from execsql.utils.gui import pause