execsql2 2.16.18__tar.gz → 2.17.0__tar.gz

{execsql2-2.16.18 → execsql2-2.17.0}/CHANGELOG.md

@@ -13,6 +13,18 @@ ______________________________________________________________________
 
 ______________________________________________________________________
 
+## [2.17.0] - 2026-05-07
+
+### Changed
+
+- **Behavior change.** `PG_UPSERT`, `PG_UPSERT QA`, and `PG_UPSERT CHECK` no longer raise a metacommand error when QA checks fail. The outcome is reported via `$PG_UPSERT_QA_PASSED` (and the per-table `$PG_UPSERT_TABLE_QA_PASSED`) along with `$PG_UPSERT_RESULT_JSON`, so the script controls flow with `IF` or `ASSERT` instead of being forced into execsql's halt-on-error path. `EXPORT_FAILURES` still runs on QA failure, and the upsert is still skipped (no commit). Scripts that previously relied on a hard halt should add `ASSERT !!$PG_UPSERT_QA_PASSED!! = TRUE` (or branch via `IF`) at the call site.
+
+### Fixed
+
+- `!!$COUNTER_N!!` references inside metacommands (e.g. `WRITE`, `IF`, `SUB`) now return the documented sequence `1, 2, 3, …` starting at 1. The AST executor was calling `substitute_vars()` twice per metacommand — once to detect a `BREAK` token before dispatch, then again inside the dispatch handler — which double-incremented every counter reference so the first read returned 2 and subsequent reads stepped by 2. Counter references in plain SQL statements were unaffected (they already substituted exactly once). The duplicate substitution also re-rolled `!!$RANDOM!!` and `!!$UUID!!` for any metacommand that referenced them, so those values are now stable across BREAK detection and dispatch as well.
+
+______________________________________________________________________
+
 ## [2.16.18] - 2026-05-05
 
 ### Fixed

{execsql2-2.16.18 → execsql2-2.17.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: execsql2
-Version: 2.16.18
+Version: 2.17.0
 Summary: Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables.
 Project-URL: Homepage, https://execsql2.readthedocs.io
 Project-URL: Repository, https://github.com/geocoug/execsql

{execsql2-2.16.18 → execsql2-2.17.0}/docs/about/divergence.md

@@ -46,15 +46,15 @@ ______________________________________________________________________
 
 ### Metacommands
 
-| Metacommand             | Description                                                                                                                                                                                                                                                                                                                                                                                                              |
-| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| `ASSERT`                | Evaluate a condition and raise an error (halting the script) if it is false. Supports all IF conditions. Optional quoted failure message. Skipped in false IF blocks.                                                                                                                                                                                                                                                    |
-| `BREAKPOINT`            | Pause script execution and drop into an interactive debug REPL. See [Debugging](#debugging) below for full details.                                                                                                                                                                                                                                                                                                      |
-| `CONFIG SHOW_PROGRESS`  | Enable the Rich progress bar for IMPORT operations at runtime.                                                                                                                                                                                                                                                                                                                                                           |
-| `CONFIG LOG_SQL`        | Enable SQL query audit logging — writes executed SQL to the log file.                                                                                                                                                                                                                                                                                                                                                    |
-| `PG_UPSERT`             | QA-checked, FK-dependency-ordered upserts from staging to base schema on PostgreSQL. Integrates [pg-upsert](https://pg-upsert.readthedocs.io/) as an optional dependency. Three modes: full pipeline, QA-only, and schema check. Supports `EXPORT_FAILURES`, `EXPORT_FORMAT`, `EXPORT_MAX_ROWS`, and `STRICT_COLUMNS` keywords. `STRICT_COLUMNS` forces all missing columns to be errors (requires `pg-upsert>=1.22.0`). |
-| `IMPORT … FROM JSON`    | Import a JSON file (array of objects or NDJSON) into a database table. Nested objects are flattened with dot-separated column names; arrays are stored as JSON strings.                                                                                                                                                                                                                                                  |
-| `SHOW SCRIPTS [<name>]` | Without a name, lists all registered SCRIPT definitions with parameter signatures and source locations. With a name, shows detail including parameters (with defaults), source file/line range, and docstring.                                                                                                                                                                                                           |
+| Metacommand             | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
+| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `ASSERT`                | Evaluate a condition and raise an error (halting the script) if it is false. Supports all IF conditions. Optional quoted failure message. Skipped in false IF blocks.                                                                                                                                                                                                                                                                                                                                                                                                  |
+| `BREAKPOINT`            | Pause script execution and drop into an interactive debug REPL. See [Debugging](#debugging) below for full details.                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
+| `CONFIG SHOW_PROGRESS`  | Enable the Rich progress bar for IMPORT operations at runtime.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+| `CONFIG LOG_SQL`        | Enable SQL query audit logging — writes executed SQL to the log file.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+| `PG_UPSERT`             | QA-checked, FK-dependency-ordered upserts from staging to base schema on PostgreSQL. Integrates [pg-upsert](https://pg-upsert.readthedocs.io/) as an optional dependency. Three modes: full pipeline, QA-only, and schema check. Supports `EXPORT_FAILURES`, `EXPORT_FORMAT`, `EXPORT_MAX_ROWS`, and `STRICT_COLUMNS` keywords. `STRICT_COLUMNS` forces all missing columns to be errors (requires `pg-upsert>=1.22.0`). QA failure does not raise — the outcome is reported via `$PG_UPSERT_QA_PASSED` so the user's own `IF` / `ON ERROR HALT` decides control flow. |
+| `IMPORT … FROM JSON`    | Import a JSON file (array of objects or NDJSON) into a database table. Nested objects are flattened with dot-separated column names; arrays are stored as JSON strings.                                                                                                                                                                                                                                                                                                                                                                                                |
+| `SHOW SCRIPTS [<name>]` | Without a name, lists all registered SCRIPT definitions with parameter signatures and source locations. With a name, shows detail including parameters (with defaults), source file/line range, and docstring.                                                                                                                                                                                                                                                                                                                                                         |
 
 ### SCRIPT Enhancements
 

{execsql2-2.16.18 → execsql2-2.17.0}/docs/reference/metacommands.md

@@ -2248,6 +2248,24 @@ Set after every `PG_UPSERT` execution:
     -- !x! WRITE `!!$PG_UPSERT_RESULT_JSON!!`
     ```
 
+### QA failure behavior
+
+`PG_UPSERT`, `PG_UPSERT QA`, and `PG_UPSERT CHECK` do **not** raise an error when QA checks fail. The outcome is reported via `$PG_UPSERT_QA_PASSED` (and the per-table `$PG_UPSERT_TABLE_QA_PASSED`), so the script controls flow with `IF` or with execsql's standard error-halt configuration. When QA fails, the metacommand still runs `EXPORT_FAILURES`, populates all `$PG_UPSERT_*` substitution variables, and skips the upsert (no commit). Pair with `ASSERT` to halt explicitly:
+
+```sql
+-- Soft failure: branch on the result, keep running.
+-- !x! PG_UPSERT FROM staging TO public TABLES books, authors EXPORT_FAILURES "qa/"
+-- !x! IF !!$PG_UPSERT_QA_PASSED!! = TRUE
+-- !x!   PG_UPSERT FROM staging TO public TABLES books, authors COMMIT
+-- !x! ELSE
+-- !x!   WRITE "QA failed — fix sheet at !!$PG_UPSERT_EXPORT_PATH!!"
+-- !x! ENDIF
+
+-- Hard failure: halt the script when QA fails.
+-- !x! PG_UPSERT QA FROM staging TO public TABLES books, authors
+-- !x! ASSERT !!$PG_UPSERT_QA_PASSED!! = TRUE "Pre-load QA failed; aborting."
+```
+
 ### Temporary objects
 
 pg-upsert creates temporary tables and views (all prefixed `ups_`) that persist after the metacommand completes. Users can query these for debugging and inspection — for example, `SELECT * FROM ups_control` shows per-table QA results and row counts.

{execsql2-2.16.18 → execsql2-2.17.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "execsql2"
-version = "2.16.18"
+version = "2.17.0"
 description = "Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables."
 readme = { file = "README.md", content-type = "text/markdown" }
 license = { file = "LICENSE.txt" }
@@ -165,7 +165,7 @@ skip-magic-trailing-comma = false
 line-ending = "auto"
 
 [tool.bumpversion]
-current_version = "2.16.18"
+current_version = "2.17.0"
 commit = true
 tag = true
 tag_name = "v{new_version}"

{execsql2-2.16.18 → execsql2-2.17.0}/src/execsql/metacommands/upsert.py

@@ -201,14 +201,6 @@ def _set_subvars(result: Any) -> None:
     sv("$PG_UPSERT_EXPORT_PATH", "")
 
 
-def _qa_failure_msg(result: Any) -> str:
-    """Build a concise QA failure message listing which tables failed."""
-    failed = [t.table_name for t in result.tables if not t.qa_passed]
-    if failed:
-        return f"PG_UPSERT QA failed for: {', '.join(failed)}"
-    return "PG_UPSERT QA checks failed."
-
-
 # ---------------------------------------------------------------------------
 # Import guard + helpers
 # ---------------------------------------------------------------------------
@@ -484,13 +476,6 @@ def x_pg_upsert(**kwargs: Any) -> None:
     if opts.get("cleanup"):
         ups.cleanup()
 
-    if not result.qa_passed:
-        raise ErrInfo(
-            "cmd",
-            command_text=metacommandline,
-            other_msg=_qa_failure_msg(result),
-        )
-
 
 def x_pg_upsert_qa(**kwargs: Any) -> None:
     """PG_UPSERT QA FROM <staging> TO <base> TABLES <t1>, <t2> [options]
@@ -524,13 +509,6 @@ def x_pg_upsert_qa(**kwargs: Any) -> None:
     if opts.get("cleanup"):
         ups.cleanup()
 
-    if not result.qa_passed:
-        raise ErrInfo(
-            "cmd",
-            command_text=metacommandline,
-            other_msg=_qa_failure_msg(result),
-        )
-
 
 def x_pg_upsert_check(**kwargs: Any) -> None:
     """PG_UPSERT CHECK FROM <staging> TO <base> TABLES <t1>, <t2>
@@ -567,13 +545,6 @@ def x_pg_upsert_check(**kwargs: Any) -> None:
     if opts.get("cleanup"):
         ups.cleanup()
 
-    if not result.qa_passed:
-        raise ErrInfo(
-            "cmd",
-            command_text=metacommandline,
-            other_msg=_qa_failure_msg(result),
-        )
-
 
 # ---------------------------------------------------------------------------
 # Plugin registration

{execsql2-2.16.18 → execsql2-2.17.0}/src/execsql/script/executor.py

@@ -243,15 +243,17 @@ def _exec_sql(
 
 def _exec_metacommand(
     ctx: RuntimeContext,
-    command: str,
+    cmd: str,
     source: str,
     line_no: int,
-    localvars: SubVarSet | None = None,
 ) -> Any:
-    """Dispatch a metacommand through the dispatch table."""
-    # Build localvars from the command-list stack frame (see _exec_sql comment).
-    effective_locals = _stack_localvars(ctx) or localvars
-    cmd = substitute_vars(command, effective_locals, ctx=ctx)
+    """Dispatch a metacommand through the dispatch table.
+
+    *cmd* must already have ``!!$VAR!!`` substitution applied.  The caller is
+    responsible for expansion so that side-effecting substitutions (counter
+    increments, ``$RANDOM``, ``$UUID``) are evaluated exactly once per
+    metacommand reference.
+    """
     if _VARLIKE.search(cmd):
         ctx.output.write(
             f"Warning: There is a potential un-substituted variable in the command\n     {cmd}\n",
@@ -355,13 +357,15 @@ def _execute_node(
         command = node.command
         if in_loop:
             command = _convert_deferred_vars(command)
-        # Intercept BREAK before dispatch — it controls loop flow
+        # Substitute once: the same expanded text is used for BREAK detection
+        # and dispatch.  Calling substitute_vars twice would double-increment
+        # !!$COUNTER_N!! and re-roll !!$RANDOM!!/!!$UUID!! references.
         effective_locals = _stack_localvars(ctx) or localvars
         expanded = substitute_vars(command, effective_locals, ctx=ctx)
         if _BREAK_RX.match(expanded):
             raise _BreakLoop
         ctx.last_command = _FakeScriptCmd(node)
-        _exec_metacommand(ctx, command, node.span.file, node.span.start_line, localvars)
+        _exec_metacommand(ctx, expanded, node.span.file, node.span.start_line)
 
     elif isinstance(node, IfBlock):
         ctx.last_command = _FakeScriptCmd(node)

{execsql2-2.16.18 → execsql2-2.17.0}/tests/metacommands/test_pg_upsert.py

@@ -19,7 +19,6 @@ from execsql.metacommands.upsert import (
     _FileWriterHandler,
     _build_result_from_qa_findings,
     _parse_tables_and_options,
-    _qa_failure_msg,
     x_pg_upsert,
     x_pg_upsert_check,
     x_pg_upsert_qa,
@@ -541,7 +540,9 @@ class TestFullMode:
         assert "$PG_UPSERT_FINISHED_AT" in calls
         assert "$PG_UPSERT_RESULT_JSON" in calls
 
-    def test_qa_failure_raises_errinfo(self, mock_state):
+    def test_qa_failure_does_not_raise(self, mock_state):
+        """QA failure surfaces via $PG_UPSERT_QA_PASSED; the user's own IF /
+        ON ERROR HALT decides control flow."""
         state, db = mock_state
         fake_result = FakeUpsertResult(
             tables=[
@@ -559,17 +560,15 @@ class TestFullMode:
             patch("execsql.metacommands.upsert._create_pgupsert") as mock_create,
         ):
             mock_create.return_value.run.return_value = fake_result
-            with pytest.raises(ErrInfo) as exc_info:
-                x_pg_upsert(
-                    staging_schema="staging",
-                    base_schema="public",
-                    tail="books",
-                    metacommandline="PG_UPSERT FROM staging TO public TABLES books",
-                )
-        err_msg = str(exc_info.value)
-        assert "QA failed for: books" in err_msg
-        # Subvars should still be set before the error
-        assert state.subvars.add_substitution.called
+            x_pg_upsert(
+                staging_schema="staging",
+                base_schema="public",
+                tail="books",
+                metacommandline="PG_UPSERT FROM staging TO public TABLES books",
+            )
+        calls = {c[0][0]: c[0][1] for c in state.subvars.add_substitution.call_args_list}
+        assert calls["$PG_UPSERT_QA_PASSED"] == "FALSE"
+        assert calls["$PG_UPSERT_COMMITTED"] == "FALSE"
 
     def test_commit_keyword_passed(self, mock_state):
         state, db = mock_state
@@ -868,42 +867,6 @@ class TestUserCancelled:
             assert "cancelled" in str(exc_info.value)
 
 
-# ---------------------------------------------------------------------------
-# Logging bridge test
-# ---------------------------------------------------------------------------
-
-
-class TestQAFailureMessage:
-    def test_lists_failed_tables(self):
-        result = FakeUpsertResult(
-            tables=[
-                FakeTableResult(
-                    table_name="books",
-                    _qa_findings=[FakeQAError(table="books", check_type="null", details="title (3)")],
-                ),
-                FakeTableResult(table_name="authors"),  # passes
-                FakeTableResult(
-                    table_name="genres",
-                    _qa_findings=[FakeQAError(table="genres", check_type="fk", details="parent_id (5)")],
-                ),
-            ],
-        )
-        msg = _qa_failure_msg(result)
-        assert msg == "PG_UPSERT QA failed for: books, genres"
-
-    def test_single_failed_table(self):
-        result = FakeUpsertResult(
-            tables=[
-                FakeTableResult(
-                    table_name="books",
-                    _qa_findings=[FakeQAError(table="books", check_type="pk", details="id (2)")],
-                ),
-            ],
-        )
-        msg = _qa_failure_msg(result)
-        assert msg == "PG_UPSERT QA failed for: books"
-
-
 # ---------------------------------------------------------------------------
 # Logging bridge tests
 # ---------------------------------------------------------------------------
@@ -934,7 +897,7 @@ class TestLoggingBridge:
 
 
 class TestQAModeFailure:
-    def test_qa_failure_raises_errinfo(self, mock_state):
+    def test_qa_failure_does_not_raise(self, mock_state):
         state, db = mock_state
 
         failed_result = FakeUpsertResult(
@@ -956,18 +919,19 @@ class TestQAModeFailure:
             mock_ups = mock_create.return_value
             mock_ups.qa_all.return_value = mock_ups
             mock_build.return_value = failed_result
-            with pytest.raises(ErrInfo) as exc_info:
-                x_pg_upsert_qa(
-                    staging_schema="staging",
-                    base_schema="public",
-                    tail="books",
-                    metacommandline="PG_UPSERT QA FROM staging TO public TABLES books",
-                )
-            assert "QA failed for: books" in str(exc_info.value)
+            x_pg_upsert_qa(
+                staging_schema="staging",
+                base_schema="public",
+                tail="books",
+                metacommandline="PG_UPSERT QA FROM staging TO public TABLES books",
+            )
+        calls = {c[0][0]: c[0][1] for c in state.subvars.add_substitution.call_args_list}
+        assert calls["$PG_UPSERT_QA_PASSED"] == "FALSE"
+        assert calls["$PG_UPSERT_COMMITTED"] == "FALSE"
 
 
 class TestCheckModeFailure:
-    def test_check_failure_raises_errinfo(self, mock_state):
+    def test_check_failure_does_not_raise(self, mock_state):
         state, db = mock_state
 
         failed_result = FakeUpsertResult(
@@ -990,14 +954,14 @@ class TestCheckModeFailure:
             mock_ups.qa_column_existence.return_value = mock_ups
             mock_ups.qa_type_mismatch.return_value = mock_ups
             mock_build.return_value = failed_result
-            with pytest.raises(ErrInfo) as exc_info:
-                x_pg_upsert_check(
-                    staging_schema="staging",
-                    base_schema="public",
-                    tail="books",
-                    metacommandline="PG_UPSERT CHECK FROM staging TO public TABLES books",
-                )
-            assert "QA failed for: books" in str(exc_info.value)
+            x_pg_upsert_check(
+                staging_schema="staging",
+                base_schema="public",
+                tail="books",
+                metacommandline="PG_UPSERT CHECK FROM staging TO public TABLES books",
+            )
+        calls = {c[0][0]: c[0][1] for c in state.subvars.add_substitution.call_args_list}
+        assert calls["$PG_UPSERT_QA_PASSED"] == "FALSE"
 
 
 # ---------------------------------------------------------------------------
@@ -1427,15 +1391,15 @@ class TestExportFailures:
             patch("execsql.metacommands.upsert._create_pgupsert") as mock_create,
         ):
             mock_create.return_value.run.return_value = fake_result
-            with pytest.raises(ErrInfo):
-                x_pg_upsert(
-                    staging_schema="staging",
-                    base_schema="public",
-                    tail="books EXPORT_FAILURES /tmp/out",
-                    metacommandline="PG_UPSERT FROM staging TO public TABLES books EXPORT_FAILURES /tmp/out",
-                )
-        # Export MUST have happened before the ErrInfo was raised.
+            x_pg_upsert(
+                staging_schema="staging",
+                base_schema="public",
+                tail="books EXPORT_FAILURES /tmp/out",
+                metacommandline="PG_UPSERT FROM staging TO public TABLES books EXPORT_FAILURES /tmp/out",
+            )
         assert fake_result.export_calls == [("/tmp/out", "csv")]
+        calls = {c[0][0]: c[0][1] for c in state.subvars.add_substitution.call_args_list}
+        assert calls["$PG_UPSERT_QA_PASSED"] == "FALSE"
 
     def test_no_export_when_keyword_absent(self, mock_state):
         state, db = mock_state

{execsql2-2.16.18 → execsql2-2.17.0}/tests/test_executor.py

@@ -154,6 +154,45 @@ class TestVariableSubstitution:
         rows = _query_db(tmp_path, "SELECT val FROM t")
         assert rows == [(8,)]
 
+    def test_counter_in_metacommand_starts_at_one(self, tmp_path):
+        # Regression: the AST executor previously called substitute_vars()
+        # twice per metacommand (once for BREAK detection, once for dispatch),
+        # which double-incremented $COUNTER_N so refs returned 2, 4, 6.
+        result = _run_ast(
+            '-- !x! WRITE "First: !!$COUNTER_1!!"\n'
+            '-- !x! WRITE "Second: !!$COUNTER_1!!"\n'
+            '-- !x! WRITE "Third: !!$COUNTER_1!!"\n',
+            tmp_path,
+        )
+        assert result.returncode == 0, result.stderr
+        assert "First: 1" in result.stdout
+        assert "Second: 2" in result.stdout
+        assert "Third: 3" in result.stdout
+
+    def test_counter_in_sql_starts_at_one(self, tmp_path):
+        # Sibling check: SQL-statement counters were already correct, but
+        # pin the behavior so a future regression in the SQL path is caught.
+        result = _run_ast(
+            "CREATE TABLE t (n INT);\n"
+            "INSERT INTO t VALUES (!!$COUNTER_2!!);\n"
+            "INSERT INTO t VALUES (!!$COUNTER_2!!);\n"
+            "INSERT INTO t VALUES (!!$COUNTER_2!!);\n",
+            tmp_path,
+        )
+        assert result.returncode == 0, result.stderr
+        rows = _query_db(tmp_path, "SELECT n FROM t ORDER BY n")
+        assert rows == [(1,), (2,), (3,)]
+
+    def test_counter_same_ref_twice_in_one_metacommand(self, tmp_path):
+        # Per docs: multiple references in one command share the same value.
+        result = _run_ast(
+            '-- !x! WRITE "A=!!$COUNTER_3!! B=!!$COUNTER_3!!"\n-- !x! WRITE "C=!!$COUNTER_3!!"\n',
+            tmp_path,
+        )
+        assert result.returncode == 0, result.stderr
+        assert "A=1 B=1" in result.stdout
+        assert "C=2" in result.stdout
+
 
 # ---------------------------------------------------------------------------
 # IF / ELSE / ELSEIF

{execsql2-2.16.18 → execsql2-2.17.0}/uv.lock

@@ -648,7 +648,7 @@ wheels = [
 
 [[package]]
 name = "execsql2"
-version = "2.16.18"
+version = "2.17.0"
 source = { editable = "." }
 dependencies = [
     { name = "python-dateutil" },