execsql2 2.5.0__tar.gz → 2.7.1__tar.gz

{execsql2-2.5.0 → execsql2-2.7.1}/.claude/project_context.md

@@ -189,10 +189,10 @@ Triggered on: push to `main`, any tag `v*.*.*`, pull requests.
 
 ## Versioning
 
-`bump-my-version` manages versions. Current: `2.4.6`. Bump commands:
+`bump-my-version` manages versions. Current: `2.5.0`. Bump commands:
 
-- `just bump-patch` → 2.4.6 → 2.4.7
-- `just bump-minor` → 2.4.6 → 2.5.0
+- `just bump-patch` → 2.5.0 → 2.5.1
+- `just bump-minor` → 2.5.0 → 2.6.0
   Bumps commit + tag. Pre-commit hook runs `uv lock` + stages `uv.lock`.
 
 ## Ruff Config
@@ -243,44 +243,132 @@ the foreseeable future.
 | Docs reorganized (getting-started/reference/guides/about) | 2.4.6  | 2026-03 |
 | Pre-commit hook for `execsql-format`              | 2.4.3          | 2026-03 |
 | VS Code syntax highlighting extension             | 2.4.x          | 2026-03 |
+| Docstring coverage 81% on public API              | 2.5.0          | 2026-04 |
+| Developer architecture guide with Mermaid diagrams | 2.5.0          | 2026-04 |
+| Cursor context managers (exec_cmd + vacuum)       | 2.5.0          | 2026-04 |
+| Exporter Protocol types (QueryExporter, RowsetExporter) | 2.5.0   | 2026-04 |
+| SubVarSet.merge() optimization (O(1) vs O(V))    | 2.5.0          | 2026-04 |
+| GitHub Actions upgraded to Node.js 24             | 2.5.0          | 2026-04 |
+| GitHub issue/PR templates + SECURITY.md           | 2.5.0          | 2026-04 |
+| Database ABC already in place (verified)          | 2.5.0          | 2026-04 |
+| Dispatch optimization already in place (verified) | 2.5.0          | 2026-04 |
+| PostgreSQL integration tests (9 tests, CI Docker) | 2.5.0          | 2026-04 |
+| MySQL integration tests (9 tests, CI Docker)      | 2.5.0          | 2026-04 |
 
 ______________________________________________________________________
 
-### v2.5 — Remaining Code Quality
+### v2.6 — Architecture & Internal Quality
 
-Theme: Complete the remaining v2.2/v2.3 items and continue stabilization.
+- [x] **`state.py` → `RuntimeContext` refactor** — 33 mutable globals consolidated into a slotted `RuntimeContext` class with transparent module proxy. `get_context()`/`set_context()` API added. Zero external call-site changes.
+- [x] **`noqa` cleanup in `metacommands/__init__.py`** — removed all 180 redundant `# noqa` comments; `__all__` already satisfies ruff F401.
+- [x] **Coverage push to 86%** — 403 new tests (3010 total) covering `db/base.py` (55→99%), `metacommands/connect.py` (36→100%), `script/engine.py` (77→95%), `exporters/delimited.py` (76→95%). Remaining gap to 90% is in GUI, ODS, and import handlers.
 
-- **Docstring coverage** — target 50%+ on public API (focus on `exporters/`, `db/`, `config.py`)
+### v2.7 — New Export/Import Formats
 
-______________________________________________________________________
+- [x] **Parquet import** — already existed as `IMPORT TO table FROM PARQUET file` (verified present).
+- [x] **YAML export** — `FORMAT YAML` via PyYAML, list-of-dicts with native type preservation.
+- [x] **Markdown (GFM) export** — `FORMAT MARKDOWN` / `MD`, pipe tables with alignment and escaping.
+- [x] **Excel (XLSX) multi-sheet export** — `FORMAT XLSX` single + multi-sheet via openpyxl, bold headers, inventory sheet, sheet name deduplication.
 
-### v2.6 — Architecture & Performance
+### v2.8 — Scripting Power Features
 
-Theme: Internal modernization. May include backward-incompatible internal API changes (public CLI behavior preserved).
+- [ ] **`ASSERT` metacommand** — `-- !x! ASSERT <condition> "message"`. Data validation for CI pipelines and sanity checks.
+- [ ] **`--dry-run` improvements** — show SQL with substitution variables expanded, not just raw metacommands.
+- [ ] **Script profiling (`--profile`)** — per-statement execution times, summary report at end. Leverages existing `Timer` infrastructure.
+- [ ] **Parallel execution blocks** — `PARALLEL BEGIN ... PARALLEL END` for independent statements. See design notes below.
 
-- **Database ABC with `@abstractmethod`** — replace runtime `DatabaseNotImplementedError` with true abstract methods
-- **Cursor context managers** — explicit cursor lifecycle in `execute()`, `select_data()`, `select_rowsource()`
-- **Metacommand dispatch optimization** — consider dict/trie lookup instead of O(N) regex scan
-- **Variable substitution optimization** — reduce O(V×D) complexity
-- **Exporter protocol** — define a `Protocol` or ABC for exporters with a common interface
+### v2.9 — Library API & Developer Experience
 
-______________________________________________________________________
+- [ ] **Programmatic Python API** — `execsql.run(script, db=...)` for notebook/pipeline usage. Depends on `RuntimeContext` refactor.
+- [ ] **TOML configuration** — `execsql.toml` as modern alternative to legacy INI format (coexist initially).
+
+### v2.10 — Testing & CI Hardening
+
+- [ ] **Property-based testing (Hypothesis)** — for parsers, type inference, substitution variables.
+- [ ] **Parser fuzzing** — `CondParser` and `NumericParser` handle arbitrary user input; fuzz for edge cases.
+- [ ] **Nightly CI against latest DB driver versions** — catch upstream breakage in psycopg2, pymysql, duckdb, etc.
+- [ ] **CI benchmarks** — track substitution variable and dispatch performance over time.
+
+### v2.11 — Documentation & Community
+
+- [ ] **Cookbook / recipes page** — real-world examples: ETL workflows, HTML reports, data validation pipelines.
+- [ ] **Migration guide from upstream execsql** — what changed, what's new, how to switch.
+- [ ] **Interactive tutorial** — guided walkthrough script against a bundled SQLite DB.
 
 ### v3.0+ — Future
 
-- **Plugin system** — allow external packages to register exporters, importers, or metacommands
+- [ ] **Plugin system** — entry points for `execsql.exporters`, `execsql.importers`, `execsql.metacommands` allowing external packages to register new handlers.
+- [ ] **LSP / language server** — for the VS Code extension: autocomplete metacommands, validate substitution variables, jump-to-definition for `INCLUDE`d scripts.
 
 ______________________________________________________________________
 
 ### Ongoing / No-milestone
 
-- PostgreSQL integration tests (requires external server — CI docker service)
-- MySQL integration tests (same)
-- `savedscripts` memory pruning
 - Textual TUI polish
 
 ______________________________________________________________________
 
+### Design Notes: Parallel Execution Blocks
+
+**Concept:** Allow users to declare groups of independent SQL statements that
+can run concurrently, reducing wall-clock time for ETL scripts with
+independent work.
+
+**Syntax:**
+```sql
+-- !x! PARALLEL BEGIN [WORKERS=4]
+INSERT INTO summary_a SELECT ... FROM raw_data;
+INSERT INTO summary_b SELECT ... FROM raw_data;
+INSERT INTO summary_c SELECT ... FROM raw_data;
+-- !x! PARALLEL END
+```
+
+**How it would work in the engine:**
+
+1. When `runscripts()` encounters `PARALLEL BEGIN`, the engine enters a
+   "collecting" mode (similar to how `LOOP` compiles commands into a
+   `CommandList` before executing). Statements are accumulated but not run.
+
+2. On `PARALLEL END`, the collected statements are dispatched to a
+   `concurrent.futures.ThreadPoolExecutor` (or `ProcessPoolExecutor`).
+   Each statement gets its own database cursor (or connection from
+   `DatabasePool`).
+
+3. The main `runscripts()` loop blocks at the `PARALLEL END` until all
+   futures complete. Errors from any worker are collected and raised as
+   a combined `ErrInfo`.
+
+**Key constraints and design decisions:**
+
+- **No shared mutable state inside parallel blocks.** Substitution variable
+  writes (`SET`), `IF/ELSE`, `LOOP`, `INCLUDE`, and other control-flow
+  metacommands are **prohibited** inside `PARALLEL` blocks — only raw SQL
+  and simple export metacommands are allowed. The parser rejects anything
+  else at compile time.
+
+- **Connection handling.** Each parallel worker needs its own cursor or
+  connection. `DatabasePool` already exists in `db/base.py` but currently
+  manages one connection per named alias. This would need a pool-per-alias
+  model (e.g., min/max connections) or each worker opens a fresh connection
+  from the same DSN.
+
+- **Depends on `RuntimeContext` refactor.** The current `state.py` globals
+  (especially `commandliststack`, `if_stack`, `subvars`) are not
+  thread-safe. Workers would need isolated read-only snapshots of
+  substitution variables and their own cursor state. The `RuntimeContext`
+  work in v2.6 is a prerequisite.
+
+- **`WORKERS=N`** defaults to `min(len(statements), os.cpu_count())`.
+  Configurable via the metacommand or `execsql.toml`.
+
+- **Transaction semantics.** Each statement runs in its own implicit
+  transaction (autocommit). If the user needs atomicity across the whole
+  block, they wrap it in `BEGIN BATCH ... END BATCH` outside the parallel
+  block — but that negates parallelism for most backends, so this is
+  mainly useful for independent ETL loads.
+
+______________________________________________________________________
+
 ## Open Design Questions
 
 ### Distribution / single-file invocation model

{execsql2-2.5.0 → execsql2-2.7.1}/.gitignore

@@ -29,6 +29,7 @@ Thumbs.db
 .pytest_cache/
 .coverage
 coverage.xml
+*.py,cover
 .tox/
 htmlcov/
 execsql.log

{execsql2-2.5.0 → execsql2-2.7.1}/CHANGELOG.md

@@ -13,7 +13,45 @@ ______________________________________________________________________
 
 ______________________________________________________________________
 
-## [2.5.0] - 2026-03-31
+## [2.7.1] - 2026-04-01
+
+### Fixed
+
+- Fix `AttributeError: module 'execsql.state' has no attribute 'dedup_words'` when importing CSV files with `DEDUP_COL_HDRS` enabled — `dedup_words` is now correctly imported from `execsql.utils.strings` instead of accessed through the state module.
+
+______________________________________________________________________
+
+## [2.7.0] - 2026-04-01
+
+### Added
+
+- **Markdown export** (`FORMAT MARKDOWN` / `FORMAT MD`) — GitHub-flavored pipe tables with column alignment, pipe/backslash escaping, and zip support. No dependencies required.
+- **YAML export** (`FORMAT YAML`) — list-of-dicts output via PyYAML with native type preservation (int, float, null). Requires `PyYAML` (included in `formats` extras).
+- **XLSX export** (`FORMAT XLSX`) — single-sheet and multi-sheet Excel export via openpyxl with bold headers, native type preservation, sheet name deduplication, and a "Datasheets" inventory sheet. Multi-sheet syntax: `EXPORT table1, table2 TO file.xlsx AS XLSX`.
+
+______________________________________________________________________
+
+## [2.6.0] - 2026-04-01
+
+### Added
+
+- Textual TUI `console_save()` — writes console output to a file, matching Tkinter parity.
+- Keyboard shortcut hints on Textual TUI dialog screens — Escape to cancel, Enter to submit, with `Footer` widget on all major dialog screens.
+- `RuntimeContext` class in `state.py` — groups all 33 mutable runtime globals into a single slotted object. Enables isolated contexts for testing and future concurrent execution.
+- `get_context()` / `set_context()` public API for programmatic access to the active runtime context.
+- Divergence from Upstream documentation page (`docs/about/divergence.md`) listing all user-visible changes since the fork.
+- Test coverage raised from 80% to 86% — 403 new tests across `db/base.py`, `metacommands/connect.py`, `script/engine.py`, and `exporters/delimited.py`.
+
+### Changed
+
+- `state.py` module now uses a `types.ModuleType` subclass that transparently proxies attribute reads and writes to the active `RuntimeContext` instance. All existing `_state.foo` call sites continue working with zero changes.
+- `reset()` simplified from 40 lines with 7 `global` statements to a clean context replacement (preserving `filewriter`).
+- `initialize()` and `endloop()` rewritten to use `_ctx` directly instead of `global` statements.
+- Removed 180 redundant `# noqa` suppressions from `metacommands/__init__.py` — the existing `__all__` already satisfies ruff F401.
+
+______________________________________________________________________
+
+## [2.5.0] - 2026-04-01
 
 ### Added
 

{execsql2-2.5.0 → execsql2-2.7.1}/CLAUDE.md

@@ -51,4 +51,5 @@ A multi-agent system where specialized agents collaborate to improve, extend, de
 - All code must pass `ruff check` and target Python 3.10+
 - **Every user-visible change must be reflected in `CHANGELOG.md`** under the `[Unreleased]` section using [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) categories: Added, Changed, Fixed, Removed. Do not leave changelog updates for later — include them in the same commit or PR as the code change.
 - **Every user-visible change must also update documentation** — review and revise `README.md`, `docs/`, and any other relevant documentation to stay consistent with the code. New CLI options, features, or behavior changes must be reflected in the README Options table, feature list, and the corresponding docs page. Do not leave doc updates for later. Docs are organized into subdirectories: `docs/getting-started/`, `docs/reference/` (configuration, metacommands, substitution_vars, security), `docs/guides/`, `docs/about/`, `docs/api/`, `docs/dev/`.
+- **Every change that diverges from upstream execsql v1.130.1 must update `docs/about/divergence.md`** — this includes new features, changed behavior, security fixes, and removed functionality. The divergence page is the canonical record of how execsql2 differs from the original monolith. Keep entries concise and organized under the existing section headings (Added Features, Changed Behavior, Security and Correctness Fixes, Removed Features).
 - **After every version bump and push, monitor CI** — push with `git push && git push --tags` to include version tags, then run `gh run list --limit 1` to get the run ID, then `gh run watch <id> --exit-status` to block until it completes. Bump commits trigger PyPI publish and GitHub Release, so failures must be caught and fixed immediately.

{execsql2-2.5.0 → execsql2-2.7.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: execsql2
-Version: 2.5.0
+Version: 2.7.1
 Summary: Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables.
 Project-URL: Repository, https://github.com/geocoug/execsql
 Project-URL: Issues, https://github.com/geocoug/execsql/issues
@@ -54,6 +54,7 @@ Requires-Dist: polars; extra == 'all'
 Requires-Dist: psycopg2-binary; extra == 'all'
 Requires-Dist: pymysql; extra == 'all'
 Requires-Dist: pyodbc; extra == 'all'
+Requires-Dist: pyyaml; extra == 'all'
 Requires-Dist: tables; extra == 'all'
 Requires-Dist: xlrd; extra == 'all'
 Provides-Extra: all-db
@@ -76,6 +77,7 @@ Requires-Dist: openpyxl; extra == 'dev'
 Requires-Dist: polars; extra == 'dev'
 Requires-Dist: pre-commit>=3.5.0; extra == 'dev'
 Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
+Requires-Dist: pyyaml; extra == 'dev'
 Requires-Dist: ruff>=0.4; extra == 'dev'
 Requires-Dist: tables; extra == 'dev'
 Requires-Dist: tox-uv>=1.13.1; extra == 'dev'
@@ -91,6 +93,7 @@ Requires-Dist: jinja2; extra == 'formats'
 Requires-Dist: odfpy; extra == 'formats'
 Requires-Dist: openpyxl; extra == 'formats'
 Requires-Dist: polars; extra == 'formats'
+Requires-Dist: pyyaml; extra == 'formats'
 Requires-Dist: tables; extra == 'formats'
 Requires-Dist: xlrd; extra == 'formats'
 Provides-Extra: mssql
@@ -227,7 +230,7 @@ Run `execsql --help` for the full option list, or `execsql -m` to list all metac
 # Features
 
 - Import data from CSV, TSV, Excel, OpenDocument, Feather, or Parquet files into a database table.
-- Export query results in 15+ formats including CSV, TSV, JSON, XML, HTML, LaTeX, OpenDocument, Feather, HDF5, DuckDB, SQLite, plain text, and Jinja2 templates.
+- Export query results in 20+ formats including CSV, TSV, JSON, YAML, XML, HTML, Markdown, LaTeX, XLSX, OpenDocument, Feather, Parquet, HDF5, DuckDB, SQLite, plain text, and Jinja2 templates.
 - Copy data between databases, including across different DBMS types.
 - Conditionally execute SQL and metacommands using `IF`/`ELSE`/`ENDIF` based on data values, DBMS type, or user input.
 - Loop over blocks of SQL and metacommands using `LOOP`/`ENDLOOP`.

{execsql2-2.5.0 → execsql2-2.7.1}/README.md

@@ -120,7 +120,7 @@ Run `execsql --help` for the full option list, or `execsql -m` to list all metac
 # Features
 
 - Import data from CSV, TSV, Excel, OpenDocument, Feather, or Parquet files into a database table.
-- Export query results in 15+ formats including CSV, TSV, JSON, XML, HTML, LaTeX, OpenDocument, Feather, HDF5, DuckDB, SQLite, plain text, and Jinja2 templates.
+- Export query results in 20+ formats including CSV, TSV, JSON, YAML, XML, HTML, Markdown, LaTeX, XLSX, OpenDocument, Feather, Parquet, HDF5, DuckDB, SQLite, plain text, and Jinja2 templates.
 - Copy data between databases, including across different DBMS types.
 - Conditionally execute SQL and metacommands using `IF`/`ELSE`/`ENDIF` based on data values, DBMS type, or user input.
 - Loop over blocks of SQL and metacommands using `LOOP`/`ENDLOOP`.

execsql2-2.7.1/docs/about/divergence.md

@@ -0,0 +1,171 @@
+# Divergence from Upstream
+
+execsql2 is a maintained fork of [execsql](https://execsql.readthedocs.io/)
+v1.130.1 by R. Dreas Nielsen. This page documents all user-visible changes
+since the fork was created: new features, changed behavior, security fixes,
+and removed functionality.
+
+For a chronological view, see the [Change Log](change_log.md).
+
+______________________________________________________________________
+
+## Added Features
+
+### CLI Options
+
+| Flag                            | Description                                                                                                                                                            |
+| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `--version`                     | Print version and exit (Rich-formatted).                                                                                                                               |
+| `-c` / `--command`              | Execute an inline SQL or metacommand string instead of a script file.                                                                                                  |
+| `--dsn` / `--connection-string` | Accept a standard database URL (e.g. `postgresql://user:pass@host/db`). Supports `postgresql`, `mysql`, `mssql`, `oracle`, `firebird`, `sqlite`, and `duckdb` schemes. |
+| `--output-dir`                  | Set a default base directory for export output files.                                                                                                                  |
+| `--progress`                    | Show a Rich progress bar during long-running IMPORT operations.                                                                                                        |
+| `--dump-keywords`               | Emit all metacommand keywords, conditionals, config options, and export formats as structured JSON.                                                                    |
+| `--gui-framework`               | Select GUI backend: `tkinter` (default) or `textual` (terminal UI).                                                                                                    |
+| `--dry-run`                     | Parse the script and print the full command list without connecting to a database or executing anything.                                                               |
+
+### Export Formats
+
+| Format            | Description                                                                                                          |
+| ----------------- | -------------------------------------------------------------------------------------------------------------------- |
+| `PARQUET`         | Export query or table results to Apache Parquet via `polars`.                                                        |
+| `FEATHER`         | Export to Apache Feather/IPC via `polars` + `pyarrow` (upstream used `pandas`).                                      |
+| `YAML`            | Export query or table results as a YAML sequence of mappings via `PyYAML`.                                           |
+| `MARKDOWN` / `MD` | Export query or table results as a GitHub-Flavored Markdown (GFM) pipe table. Pure Python, no optional dependencies. |
+| `XLSX`            | Export query or table results to an Excel XLSX workbook via `openpyxl` (single or multi-sheet).                      |
+
+### Metacommands
+
+| Metacommand            | Description                                                           |
+| ---------------------- | --------------------------------------------------------------------- |
+| `CONFIG SHOW_PROGRESS` | Enable the Rich progress bar for IMPORT operations at runtime.        |
+| `CONFIG LOG_SQL`       | Enable SQL query audit logging — writes executed SQL to the log file. |
+
+### Configuration Options
+
+New options in `execsql.conf`:
+
+| Option                     | Section     | Description                                                       |
+| -------------------------- | ----------- | ----------------------------------------------------------------- |
+| `use_keyring`              | `[connect]` | Use the OS keyring for credential storage (default: `yes`).       |
+| `show_progress`            | `[input]`   | Enable Rich progress bar for IMPORT (default: `no`).              |
+| `import_progress_interval` | `[input]`   | Log a status line every N rows during IMPORT (default: `0`).      |
+| `log_sql`                  | `[config]`  | Enable SQL audit logging (default: `no`).                         |
+| `max_log_size_mb`          | `[config]`  | Rotate the log file at this size in MB (default: `0` = disabled). |
+
+### Tools
+
+| Tool             | Description                                                                                                                                                                                    |
+| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `execsql-format` | Standalone CLI for normalizing metacommand indentation and uppercasing SQL keywords. Supports `--check` and `--in-place` modes. Also available as a [pre-commit hook](../guides/formatter.md). |
+
+### GUI
+
+| Feature             | Description                                                                                                                                             |
+| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Textual TUI backend | Full terminal-UI backend via the `textual` library. Provides all dialog types (password, pause, message, entry, compare, action, etc.) in the terminal. |
+| Console fallback    | Text-only backend that handles GUI calls in headless environments by printing to stdout.                                                                |
+
+### Authentication
+
+| Feature                       | Description                                                                                                                                                                    |
+| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| OS keyring integration        | When the `keyring` package is installed, passwords are stored in and retrieved from the OS credential store (macOS Keychain, Windows Credential Manager, Linux SecretService). |
+| Keyring retry on auth failure | If a stored password is rejected, the stale entry is deleted, the user is re-prompted, and the new password is saved automatically.                                            |
+
+### Logging Enhancements
+
+| Feature                       | Description                                                                        |
+| ----------------------------- | ---------------------------------------------------------------------------------- |
+| Per-event ISO 8601 timestamps | `status`, `connect`, `action`, and `user_msg` log entries include a timestamp.     |
+| Run duration in exit record   | The `exit` log record includes elapsed wall-clock time.                            |
+| Run ID millisecond precision  | Run identifier format changed from `%Y%m%d_%H%M_%S` to `%Y%m%d_%H%M_%S_NNN`.       |
+| SQL audit log record          | New `sql` record type containing DB name, line number, and query text.             |
+| Import progress log           | Periodic row-count status lines during IMPORT when `import_progress_interval > 0`. |
+
+### Developer / Packaging
+
+| Feature                     | Description                                                                                                     |
+| --------------------------- | --------------------------------------------------------------------------------------------------------------- |
+| VS Code syntax highlighting | Auto-generated `tmLanguage.json` grammar from the dispatch table.                                               |
+| `py.typed` marker           | PEP 561 marker enabling downstream static type checking.                                                        |
+| Structured keyword registry | `--dump-keywords` introspects the dispatch table and outputs JSON used by the grammar generator and test suite. |
+
+______________________________________________________________________
+
+## Changed Behavior
+
+### CLI Interface
+
+The CLI framework changed from `optparse` to [Typer](https://typer.tiangolo.com/) with Rich-formatted help text. All original short flags (`-a` through `-z`) are preserved. The tool can be invoked as either `execsql` or `execsql2`.
+
+### Internal State Management
+
+All 33 mutable runtime globals in `state.py` have been consolidated into a `RuntimeContext` object. The module uses a transparent proxy so existing code is unaffected, but the architecture now supports isolated contexts for testing and future concurrent execution.
+
+### Substitution Variables
+
+- **Cycle detection** — `substitute_vars()` raises an error after 100 iterations to prevent infinite loops when variables reference each other cyclically. Upstream had no protection.
+- **O(1) substitution** — Variable substitution uses a single combined regex and dict lookup instead of O(V) per-variable regex passes. Behavior is identical; performance is improved.
+
+### Database Adapters
+
+- **`Database` is an ABC** — `open_db()` and `exec_cmd()` are abstract methods. Subclasses that omit them raise `TypeError` at instantiation instead of at call time.
+- **Connection timeouts** — PostgreSQL and SQLite adapters accept a connection timeout parameter (default 30 seconds).
+- **DuckDB temporal types** — `TIMESTAMPTZ`, `TIMESTAMP`, `DATE`, `TIME` now map to native DuckDB types instead of `TEXT`.
+
+### Error Handling
+
+- **Exception hierarchy** — All custom exceptions inherit from `ExecSqlError`, enabling `except ExecSqlError` to catch any execsql-originated error.
+- **Exception chaining** — All `raise` statements inside `except` blocks preserve the original traceback via `from`.
+
+______________________________________________________________________
+
+## Security and Correctness Fixes
+
+These are behavioral changes driven by security or correctness issues in the upstream code.
+
+### Injection Fixes
+
+| Area                         | Fix                                                                                                                                                                                                  |
+| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Database metadata queries    | `schema_exists()`, `table_exists()`, `column_exists()`, `table_columns()`, `view_exists()`, `role_exists()` across all 9 adapters now use parameterized queries. Upstream used string interpolation. |
+| `import_entire_file()`       | Column names are quoted with `quote_identifier()` instead of interpolated into INSERT statements.                                                                                                    |
+| PostgreSQL `CREATE DATABASE` | Database name and encoding are quoted. COPY delimiter and quote character are validated.                                                                                                             |
+| `$SHEETS_TABLES_VALUES`      | Sheet names from ODS/XLS imports are escaped before embedding in SQL.                                                                                                                                |
+| HTTP `Content-Disposition`   | Filename is sanitized to prevent HTTP response splitting in SERVE.                                                                                                                                   |
+
+### Template and Export Safety
+
+| Area              | Fix                                                                                                    |
+| ----------------- | ------------------------------------------------------------------------------------------------------ |
+| Jinja2 sandboxing | Templates run in `SandboxedEnvironment` instead of the default `jinja2.Template`.                      |
+| HTML export       | Column headers and cell values are escaped with `html.escape()` to prevent XSS.                        |
+| XML export        | Values are escaped with `xml.sax.saxutils.escape()`. Invalid XML element name characters are replaced. |
+| JSON export       | The `description` field uses `json.dumps()` instead of string interpolation.                           |
+
+### Credential and Logging Safety
+
+| Area                         | Fix                                                                                        |
+| ---------------------------- | ------------------------------------------------------------------------------------------ |
+| ODBC password redaction      | Connection strings in log output have `Pwd=***` substituted before logging.                |
+| `enc_password` documentation | Prominent warnings that XOR encryption is obfuscation only — keys are hardcoded in source. |
+
+### Bug Fixes
+
+| Area                              | Fix                                                                                                        |
+| --------------------------------- | ---------------------------------------------------------------------------------------------------------- |
+| Oracle default port               | Corrected from `5432` (PostgreSQL) to `1521`.                                                              |
+| MySQL `LOAD DATA INFILE` encoding | Python encoding names are now mapped to MySQL charset names.                                               |
+| `dt_cast` type converters         | Base `Database` class auto-populates 8 type converters that were previously left empty after the refactor. |
+| `FileWriter` CPU busy-loop        | Uses blocking `queue.get(timeout=0.1)` instead of `get_nowait()` in a tight loop.                          |
+| Substitution variable cycles      | 100-iteration limit prevents infinite loops on cyclic variable references.                                 |
+
+______________________________________________________________________
+
+## Removed Features
+
+| Feature                     | Reason                                                                                                                                                                                  |
+| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Airspeed template processor | The `airspeed` library (Velocity clone) is unmaintained since ~2018. Use `FORMAT jinja` instead. The `airspeed` value for `template_processor` in `execsql.conf` is no longer accepted. |
+| Python 2 compatibility      | All Python 2 constructs (`stringtypes`, `u""` literals, `optparse`, etc.) have been removed. execsql2 requires Python 3.10+.                                                            |

{execsql2-2.5.0 → execsql2-2.7.1}/docs/reference/metacommands.md

@@ -932,7 +932,22 @@ JSON_TS or JSON_TABLESCHEMA
 
 LATEX
 
-:   Input for the [LaTeΧ](https://www.latex-project.org/) typesetting system. If the "APPEND" keyword is not used, a complete document (of class article) will be written. If the "APPEND" keyword is used, only the table definition will be written to the output file. If the "APPEND" keyword is used and an existing output file contains an \\end directive, the table will be written before that directive rather than at the physical end of the file. Wide or long tables may exceed LaTeΧ's default page size. If the "DESCRIPTION" keyword is used, the given description will be used as the table's caption. Data exported in LaTeX format cannot be written into a zipfile.
+:   Input for the [LaTeX](https://www.latex-project.org/) typesetting system. If the "APPEND" keyword is not used, a complete document (of class article) will be written. If the "APPEND" keyword is used, only the table definition will be written to the output file. If the "APPEND" keyword is used and an existing output file contains an \\end directive, the table will be written before that directive rather than at the physical end of the file. Wide or long tables may exceed LaTeΧ's default page size. If the "DESCRIPTION" keyword is used, the given description will be used as the table's caption. Data exported in LaTeX format cannot be written into a zipfile.
+
+
+MARKDOWN or MD
+
+:   [GitHub-Flavored Markdown](https://github.github.com/gfm/) pipe table. Column values are aligned and pipe (`|`) and backslash (`\`) characters in data are escaped. If the "DESCRIPTION" keyword is used, the description is written as an HTML comment (`<!-- ... -->`) before the table. If the "APPEND" keyword is used, only the table is appended (no repeated headers). No optional dependencies required.
+
+
+XLSX
+
+:   [Excel](https://www.microsoft.com/en-us/microsoft-365/excel) workbook in the Office Open XML format. One or more tables (or views) can be exported to an XLSX workbook. Each table will be exported to a separate worksheet within the workbook, with the first row containing bold column headers. To export multiple tables, their names must be separated by commas. The "APPEND" keyword can be used to add worksheets to an existing workbook. The name of the view or table exported will be used as the worksheet name; if this conflicts with a sheet already in the workbook, a number will be appended to make the sheet name unique. A "Datasheets" inventory sheet is created with author, date, description, and source information for each data sheet. Data types are preserved natively (integers, floats, dates, datetimes, booleans). The `openpyxl` library must be installed (`pip install execsql2[excel]`). Data exported in XLSX format cannot be written into a zipfile.
+
+
+YAML
+
+:   [YAML](https://yaml.org/) sequence of mappings. Each row is represented as a mapping (dictionary) with column names as keys. Python data types are preserved — integers remain integers, floats remain floats, and `None` becomes YAML `null`. If the "APPEND" keyword is used, a new YAML document is appended to the file (multi-document stream). The `PyYAML` library must be installed (`pip install execsql2[formats]`). No description text is included in the output even if provided.
 
 
 SQLITE

{execsql2-2.5.0 → execsql2-2.7.1}/extras/vscode-execsql/syntaxes/execsql.tmLanguage.json

@@ -118,7 +118,7 @@
     },
     "export-formats": {
       "comment": "Export/import format names",
-      "match": "(?i)\\b(json_tableschema|cgi-html|feather|json_ts|txt-and|unitsep|duckdb|sqlite|values|latex|plain|hdf5|html|json|tabq|tsvq|b64|csv|ods|raw|tab|tsv|txt|xml|us)\\b",
+      "match": "(?i)\\b(json_tableschema|cgi-html|markdown|feather|json_ts|txt-and|unitsep|duckdb|sqlite|values|latex|plain|hdf5|html|json|tabq|tsvq|xlsx|yaml|b64|csv|ods|raw|tab|tsv|txt|xml|md|us)\\b",
       "name": "support.constant.execsql"
     },
     "secondary-operators": {

{execsql2-2.5.0 → execsql2-2.7.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "execsql2"
-version = "2.5.0"
+version = "2.7.1"
 description = "Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables."
 readme = { file = "README.md", content-type = "text/markdown" }
 license = { file = "LICENSE.txt" }
@@ -56,7 +56,7 @@ firebird = ["firebird-driver"]
 oracle = ["oracledb"]
 odbc = ["pyodbc"]
 # Feature bundles
-formats = ["odfpy", "xlrd", "openpyxl", "Jinja2", "polars", "tables"]
+formats = ["odfpy", "xlrd", "openpyxl", "Jinja2", "polars", "tables", "PyYAML"]
 auth = ["keyring"]
 # Convenience groups
 all-db = [
@@ -147,6 +147,10 @@ ignore = [
     "UP017",  # Use datetime.UTC alias (not available in Python 3.10)
 ]
 
+[tool.ruff.lint.per-file-ignores]
+# F822: context attrs are proxied via __getattr__/__setattr__ on the module class, not defined as module-level variables
+"src/execsql/state.py" = ["F822"]
+
 [tool.ruff.format]
 quote-style = "double"
 indent-style = "space"
@@ -154,7 +158,7 @@ skip-magic-trailing-comma = false
 line-ending = "auto"
 
 [tool.bumpversion]
-current_version = "2.5.0"
+current_version = "2.7.1"
 commit = true
 commit_args = "--no-verify"
 tag = true
@@ -178,7 +182,7 @@ addopts = [
     "--cov=execsql",
     "--cov-branch",
     "--cov-report=xml",
-    "--cov-fail-under=80",
+    "--cov-fail-under=85",
     "--strict-markers",
     "--strict-config",
     "--color=yes",

{execsql2-2.5.0 → execsql2-2.7.1}/src/execsql/exporters/__init__.py

@@ -9,9 +9,9 @@ handlers can access them via ``_state.write_query_to_csv`` etc. without
 importing directly from here.
 
 Sub-modules: ``base``, ``delimited``, ``json``, ``xml``, ``html``,
-``latex``, ``ods``, ``xls``, ``zip``, ``raw``, ``pretty``, ``values``,
-``templates``, ``feather``, ``parquet``, ``duckdb``, ``sqlite``,
-``protocol``.
+``latex``, ``markdown``, ``ods``, ``xls``, ``zip``, ``raw``, ``pretty``,
+``values``, ``templates``, ``feather``, ``parquet``, ``duckdb``,
+``sqlite``, ``protocol``.
 """
 
 from execsql.exporters.protocol import QueryExporter, RowsetExporter

{execsql2-2.5.0 → execsql2-2.7.1}/src/execsql/exporters/delimited.py

@@ -27,7 +27,7 @@ from execsql.exceptions import ErrInfo
 from execsql.models import DataTable
 from execsql.utils.errors import exception_desc
 from execsql.utils.fileio import filewriter_close
-from execsql.utils.strings import clean_words, fold_words
+from execsql.utils.strings import clean_words, dedup_words, fold_words
 
 __all__ = ["LineDelimiter", "CsvFile", "CsvWriter", "DelimitedWriter", "write_delimited_file"]
 
@@ -677,7 +677,7 @@ class CsvFile(EncodedFile):
         if conf.fold_col_hdrs != "no":
             colnames = fold_words(colnames, conf.fold_col_hdrs)
         if conf.dedup_col_hdrs:
-            colnames = _state.dedup_words(colnames)
+            colnames = dedup_words(colnames)
         return colnames
 
     def column_headers(self) -> list[str]: