execsql2 2.1.1__tar.gz → 2.2.1__tar.gz

{execsql2-2.1.1 → execsql2-2.2.1}/.claude/project_context.md

@@ -7,6 +7,8 @@ ______________________________________________________________________
 > **Canonical location:** `.claude/project_context.md` in the repo.
 > Global memory (`~/.claude/projects/.../memory/`) mirrors this — the repo file is the source of truth.
 > Update this file whenever any architectural, tooling, or directional decision is made.
+> Update the CHANGELOG.md whenever making a release or significant change.
+> Update ROADMAP.md whenever making a release or significant change (if direction changes or if features have already been implemented)
 
 ______________________________________________________________________
 
@@ -78,9 +80,17 @@ src/execsql/               # active codebase — all new work goes here
     auth.py / crypto.py / datetime.py / errors.py / fileio.py
     gui.py / mail.py / numeric.py / regex.py / strings.py / timer.py
 docs/               # 18 MkDocs Markdown pages (Material theme)
+extras/
+  vscode-execsql/   # VS Code syntax highlighting extension (repo-only, not in wheel)
+    package.json
+    syntaxes/
+      execsql.tmLanguage.json  # auto-generated via `just generate-vscode-grammar`
+scripts/
+  generate_vscode_grammar.py   # generates tmLanguage.json from --dump-keywords
 templates/          # SQL templates + config files
 tests/
   test_placeholder.py
+  test_registry.py  # keyword consistency tests (dispatch table ↔ grammar ↔ CLI)
 .github/workflows/
   ci-cd.yml         # CI/CD pipeline (see below)
 .pre-commit-config.yaml
@@ -97,18 +107,18 @@ ______________________________________________________________________
 
 ## Tooling Decisions
 
-| Tool                | Purpose                  | Decision                                                                     |
-| ------------------- | ------------------------ | ---------------------------------------------------------------------------- |
-| `uv`                | Package/env management   | Chosen over pip/poetry                                                       |
-| `hatchling`         | Build backend            | via pyproject.toml                                                           |
-| `ruff`              | Lint + format            | Permissive config during migration, tighten during refactor                  |
-| `tox-uv`            | Multi-Python test matrix | py310–py313                                                                  |
-| `bump-my-version`   | Version bumping          | tags + commits, hooks run `uv lock`                                          |
-| `just`              | Task runner              | `lint`, `test`, `test-all`, `docs`, `docs-serve`, `bump-patch`, `bump-minor` |
-| `pre-commit`        | Git hooks                | gitleaks, uv-lock, ruff, mdformat, markdownlint, typos, validate-pyproject   |
-| `mkdocs` + Material | Docs site                | converted from Sphinx/RST                                                    |
-| `mkdocstrings`      | API docs from docstrings | installed, not yet wired in                                                  |
-| `pytest-cov`        | Coverage                 | configured, `--cov-fail-under` commented out until tests are written         |
+| Tool                | Purpose                  | Decision                                                                                                |
+| ------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------- |
+| `uv`                | Package/env management   | Chosen over pip/poetry                                                                                  |
+| `hatchling`         | Build backend            | via pyproject.toml                                                                                      |
+| `ruff`              | Lint + format            | Permissive config during migration, tighten during refactor                                             |
+| `tox-uv`            | Multi-Python test matrix | py310–py313                                                                                             |
+| `bump-my-version`   | Version bumping          | tags + commits, hooks run `uv lock`                                                                     |
+| `just`              | Task runner              | `lint`, `test`, `test-all`, `docs`, `docs-serve`, `bump-*`, `generate-vscode-grammar`, `install-vscode` |
+| `pre-commit`        | Git hooks                | gitleaks, uv-lock, ruff, mdformat, markdownlint, typos, validate-pyproject                              |
+| `mkdocs` + Material | Docs site                | converted from Sphinx/RST                                                                               |
+| `mkdocstrings`      | API docs from docstrings | installed, not yet wired in                                                                             |
+| `pytest-cov`        | Coverage                 | configured, `--cov-fail-under` commented out until tests are written                                    |
 
 ## Package Layout Decision
 
@@ -154,72 +164,97 @@ Triggered on: push to `main`, any tag `v*.*.*`, pull requests.
 
 `target-version = "py313"`, `line-length = 120`. Currently permissive — many legacy rules ignored (tabs, bare except, ambiguous names, unused imports, etc.). Intent is to tighten rules progressively during refactor.
 
-## Known Issues / Docs Debt
+## Keyword Registry & VS Code Extension
 
-*(Docs cleanup completed — anchor IDs fixed, RST artifacts resolved.)*
+The dispatch table is the single source of truth for all metacommand keywords. Every `mcl.add()` call in `metacommands/__init__.py` has `description=` (keyword name) and `category=` (one of: `control`, `block`, `action`, `config`, `config_option`, `prompt`). Conditional functions in `conditions.py` use `category="condition"`.
 
-## Roadmap / Open Work
+- `execsql --dump-keywords` introspects the dispatch table at runtime and outputs structured JSON
+- `scripts/generate_vscode_grammar.py` consumes that JSON to produce `extras/vscode-execsql/syntaxes/execsql.tmLanguage.json`
+- `tests/test_registry.py` validates keyword consistency across dispatch table, CLI output, and grammar
+- Export format constants (`QUERY_EXPORT_FORMATS`, `TABLE_EXPORT_FORMATS`, etc.) are centralized in `metacommands/__init__.py`
+- The VS Code extension lives at `extras/vscode-execsql/` — it is NOT included in the Python wheel (it's an editor extension, not a library). Install via `just install-vscode` (symlinks into `~/.vscode/extensions/`).
 
-Items are ordered by recommended priority. Each has a status and a brief
-rationale so the right one can be picked up without re-investigating context.
+**When adding a new metacommand:** add `description=` and `category=` to the `mcl.add()` call, then run `just generate-vscode-grammar`.
 
 ______________________________________________________________________
 
-### 1. Orphaned optional features ✅ *completed 2026-03-23*
+## Known Issues / Docs Debt
 
-- **Airspeed removed:** `AirspeedTemplateReport` and `FORMAT airspeed` deleted from
-    `exporters/templates.py`; `airspeed` removed from valid `template_processor` config values.
-    Noted in `CHANGELOG.md` under `[Unreleased]`.
-- **`feather = ["pandas", "pyarrow"]`** added to `pyproject.toml` optional-deps and `all` group.
-- **`hdf5 = ["tables"]`** added to `pyproject.toml` optional-deps and `all` group.
+*(Docs cleanup completed — anchor IDs fixed, RST artifacts resolved.)*
 
-______________________________________________________________________
+## Roadmap
 
-### 2. Wire `mkdocstrings` into the docs *(low effort, high payoff)*
+Milestones are v2.x minor releases. Python 3.10 support is maintained for
+the foreseeable future. See also `ROADMAP.md` in the repo root.
 
-`mkdocstrings-python` is installed as a dev dep but no `:::` autodoc directives
-exist in any docs page. The modular refactor means every public class and
-function now has a docstring. Adding API reference pages would give users
-discoverable, always-up-to-date docs from zero extra writing.
+### Completed
 
-**Recommended action:** Add an `api/` section to the MkDocs nav with at least
-one page per top-level module group (`db/`, `exporters/`, `importers/`,
-`metacommands/`). Start with the most user-facing ones (`db/base.py`,
-`exporters/`, `cli.py`).
+| Item                                              | Version       | Date    |
+| ------------------------------------------------- | ------------- | ------- |
+| Monolith → modular refactor (80+ files)           | 2.0.0a1–2.1.0 | 2026-03 |
+| 30+ security/correctness fixes (ANALYSIS.md)      | 2.1.x         | 2026-03 |
+| mkdocstrings API docs wired up                    | 2.1.0         | 2026-03 |
+| `execsql-format` surfaced in README + docs nav    | 2.1.0         | 2026-03 |
+| Coverage floor raised to 70% (2,055 tests)        | 2.1.2         | 2026-03 |
+| Orphaned optional features cleaned up             | 2.1.0         | 2026-03 |
+| Keyring auth, Parquet/Feather/HDF5 export, DuckDB | 2.1.0         | 2026-03 |
+| Progress bars, SQL audit logging                  | unreleased    | 2026-03 |
+| Full monolith parity verified                     | unreleased    | 2026-03 |
 
 ______________________________________________________________________
 
-### 3. Surface `execsql-format` in the README and docs nav ✅ *already complete*
+### v2.2 — Stability & Testing
+
+Theme: Harden what exists before adding features.
 
-- README has a "Formatting Scripts" section (lines 95–107) with usage examples and a docs link.
-- `zensical.toml` nav lists `{ "execsql-format" = "formatter.md" }` under Guides.
-- Completed in commit `a25d1f4`.
+- **SQLite integration tests** — full lifecycle (connect, DDL, CRUD, export/import, disconnect)
+- **End-to-end CLI tests** — run scripts through the actual CLI entry point
+- **Exception chaining** — add `from e` to ~80 `except Exception: raise ErrInfo(...)` patterns
+- **Exception hierarchy** — make `ErrInfo`, `DataTypeError`, `DbTypeError`, `DatabaseNotImplementedError` inherit from `ExecSqlError`
+- **`py.typed` marker** — add `src/execsql/py.typed` for downstream type checking
+- **README accuracy** — remove "not yet stable" warning, update feature list
+- **Raise coverage floor** — target 75%
 
 ______________________________________________________________________
 
-### 4. Raise the coverage floor *(follows test work)* ✅ *floor at 68% as of 2026-03-24*
+### v2.3 — Code Quality & Documentation
 
-Coverage is at 68.18% as of 2026-03-24 with the floor at 68% (`--cov-fail-under=68`). The next
-natural targets to push further:
+Theme: Make the codebase welcoming to contributors and transparent to users.
 
-- `importers/{ods,xls,feather}` — same pattern as the exporter tests already written
-- `metacommands/` handlers not yet exercised end-to-end (EXPORT SQLITE/DUCKDB via CLI)
-- `db/duckdb.py` deeper methods (mirrors what was done for `db/sqlite.py`)
+- **Docstring coverage** — target 50%+ on public API (focus on `db/`, `exporters/`, `script.py`, `config.py`)
+- **CONTRIBUTING.md** — dev setup, architecture overview, PR workflow, testing expectations
+- **Security documentation** — document trust model and security boundaries (SHELL, path traversal, SMTP, passwords)
+- **Ruff tightening** — enable E722 (bare except), E721 (type comparison), remove legacy suppressions
+- **`__all__` exports** — add to all public modules
+- **Remove lazy import anti-pattern** — replace ~30 `global module; import module` with top-level or proper lazy imports
 
-**Recommended action:** Write the importer tests first (straightforward),
-then raise `--cov-fail-under` from 68 → 70+.
+______________________________________________________________________
+
+### v2.4 — Architecture & Performance
+
+Theme: Internal modernization. May include backward-incompatible internal API changes (public CLI behavior preserved).
+
+- **Split large modules** — `script.py` (1,157 lines) and `metacommands/__init__.py` (1,606 lines)
+- **Database ABC with `@abstractmethod`** — replace runtime `DatabaseNotImplementedError` with true abstract methods
+- **Cursor context managers** — explicit cursor lifecycle in `execute()`, `select_data()`, `select_rowsource()`
+- **Metacommand dispatch optimization** — consider dict/trie lookup instead of O(N) regex scan
+- **Variable substitution optimization** — reduce O(V×D) complexity
+- **Exporter protocol** — define a `Protocol` or ABC for exporters with a common interface
 
 ______________________________________________________________________
 
-### 5. Ruff tightening *(ongoing, lowest urgency)*
+### v2.5+ — Future
+
+- **Plugin system** — allow external packages to register exporters, importers, or metacommands
+
+______________________________________________________________________
 
-Progressive modernization: enable stricter Ruff rules, remove Py2 compat
-remnants (`type(x) == y` → `isinstance`, bare `except:` → `except Exception:`,
-etc.), modernize idioms. No user-facing change.
+### Ongoing / No-milestone
 
-**Recommended action:** Work rule-by-rule. Start by removing the most
-egregious suppressions in `pyproject.toml` (`E722` bare-except, `E721`
-type-comparison). Fix each batch of violations before enabling the next rule.
+- PostgreSQL integration tests (requires external server — CI docker service)
+- MySQL integration tests (same)
+- `savedscripts` memory pruning
+- Textual TUI polish
 
 ______________________________________________________________________
 

{execsql2-2.1.1 → execsql2-2.2.1}/.github/workflows/ci-cd.yml

@@ -50,6 +50,7 @@ jobs:
                   python -m pip install --upgrade pip
                   python -m pip install tox
                   python -m pip install ".[dev]"
+                  python -m pip install ".[formats]"
             - name: Install tkinter (Ubuntu)
               if: runner.os == 'Linux'
               run: sudo apt-get install -y python3-tk

{execsql2-2.1.1 → execsql2-2.2.1}/.gitignore

@@ -31,7 +31,12 @@ coverage.xml
 .tox/
 htmlcov/
 execsql.log
-scripts/
+scripts/execsql.conf
+scripts/*.csv
+scripts/test_import_sqlite.sql
+scripts/test_script.sql
+scripts/test.db
+scripts/*.log
 
 # Distribution
 *.whl

{execsql2-2.1.1 → execsql2-2.2.1}/.pre-commit-config.yaml

@@ -18,7 +18,7 @@ repos:
       - id: gitleaks
   # uv lockfile management
   - repo: https://github.com/astral-sh/uv-pre-commit
-    rev: 0.10.12
+    rev: 0.11.1
     hooks:
       - id: uv-lock
   # Base hooks
@@ -47,6 +47,7 @@ repos:
           - mdformat-mkdocs==5.1.1
         exclude: |
           (?x)^(
+            docs/api/.*|
             docs/metacommands\.md|
             docs/documentation\.md|
             docs/examples\.md

{execsql2-2.1.1 → execsql2-2.2.1}/.readthedocs.yaml

@@ -9,7 +9,8 @@ build:
     python: "3.13"
   jobs:
     install:
-      - pip install zensical
+      - pip install zensical mkdocstrings-python
+      - pip install -e .
     pre_build:
       - cp CHANGELOG.md docs/change_log.md
     build:

{execsql2-2.1.1 → execsql2-2.2.1}/CHANGELOG.md

@@ -13,6 +13,134 @@ ______________________________________________________________________
 
 ______________________________________________________________________
 
+## [2.2.1] - 2026-03-26
+
+### Fixed
+
+- Skip `TimerHandler` alarm tests on Windows where `signal.setitimer` is unavailable.
+- Fix `UnicodeDecodeError` in CLI subprocess tests on Windows by specifying UTF-8 encoding.
+
+______________________________________________________________________
+
+## [2.2.0] - 2026-03-26
+
+### Added
+
+- `py.typed` marker for PEP 561 downstream type checking.
+- 252 new tests (2,062 → 2,314) covering metacommands (`data`, `system`, `io_fileops`, `io_write`), utils (`auth`, `errors`, `fileio`, `timer`), and SQLite integration tests for JSON/HTML/LaTeX/TSV exports, DDL operations, WRITE-to-file, CONFIG metacommand, error handling, and inline commands; end-to-end CLI tests for `-c`, `--dsn sqlite://`, `-a` substitution, `--dry-run`, `--dump-keywords`, and `--version`. Coverage floor raised from 70% to 75%.
+- `ods` optional-dependency extra in `pyproject.toml` (`pip install execsql2[ods]`).
+- Keyring credential storage documentation in usage notes.
+- `--progress` CLI flag and `CONFIG SHOW_PROGRESS` metacommand to display a rich progress bar during long-running IMPORT operations. Also configurable via `show_progress` in `execsql.conf`. (FEAT-5)
+- Opt-in SQL query audit logging via `log_sql` config option and `CONFIG LOG_SQL` metacommand. When enabled, all executed SQL statements are written to the log file with a `sql` record type, database name, line number, and query text. (FEAT-6)
+- `--dump-keywords` CLI option: outputs all metacommand keywords, conditional functions, config options, export formats, database types, and variable patterns as structured JSON. Enables tooling (e.g., editor grammar generators) to consume keyword data directly from the dispatch table.
+- VS Code syntax highlighting extension colocated at `extras/vscode-execsql/`. The grammar (`execsql.tmLanguage.json`) is auto-generated from the dispatch table via `just generate-vscode-grammar`.
+- Keyword registry: `MetaCommand` and `MetaCommandList` now support `category` parameter. All `mcl.add()` calls are tagged with `description=` and `category=`, making the dispatch table the single source of truth for keyword metadata.
+- Export format constants (`QUERY_EXPORT_FORMATS`, `TABLE_EXPORT_FORMATS`, `SERVE_FORMATS`, etc.) centralized in `metacommands/__init__.py` and used in dispatch table regex construction.
+- `tests/test_registry.py`: keyword consistency tests validating `--dump-keywords` output, dispatch table categories, conditional table coverage, export format constants, and grammar synchronization.
+
+### Changed
+
+- CLI reorganized from flat `_cli_*.py` files into `cli/` subpackage (`cli/__init__.py`, `cli/run.py`, `cli/dsn.py`, `cli/help.py`). All existing `from execsql.cli import ...` paths preserved.
+- Exception hierarchy: `ErrInfo`, `DataTypeError`, `DbTypeError`, and `DatabaseNotImplementedError` now inherit from `ExecSqlError` base class instead of bare `Exception`.
+- Exception chaining: added `from e` to 115 `raise` statements across 38 files that previously discarded the original exception context.
+- README: replaced "not yet stable" warning with "maintained fork" note reflecting current project maturity.
+- Split `cli.py` (1245 lines) into `_cli_help.py`, `_cli_dsn.py`, and `_cli_run.py` with `cli.py` as a re-export façade. All existing import paths preserved. (REFAC-3)
+- Split `metacommands/io.py` (1304 lines) into `io_export.py`, `io_import.py`, `io_write.py`, and `io_fileops.py` with `io.py` as a re-export façade. All existing import paths preserved. (REFAC-4)
+
+### Fixed
+
+- `of` module typo throughout `exporters/ods.py`: all imports and references used `of` (e.g., `import of as of`, `of.table.Table`) instead of `of` (e.g., `import of as of`, `of.table.Table`). ODS export/import was completely broken when `odfpy` was installed. Same typo fixed in `tests/exporters/test_ods.py` and `tests/importers/test_ods_importer.py`.
+
+- Unclosed `ScriptFile` in `script.py`: `read_sqlfile()` iterated through the file but never closed it, leaking a file handle on every script load.
+
+- Unclosed CSV file handle in `exporters/delimited.py`: `evaluate_line_format()` opened a file for delimiter diagnosis but never closed it. `reader()` generator also leaked the file handle if not fully consumed or if an exception occurred mid-iteration. Both now use `try/finally`.
+
+- Unclosed `sqlite3.Connection` in test assertions: `with sqlite3.connect(...)` only commits/rolls back but does not close the connection. Fixed in `tests/exporters/test_sqlite_exporter.py` and `tests/metacommands/test_metacommands.py`.
+
+- File handle leaks in exporters: wrapped write operations in `try/finally` blocks to guarantee file handles are closed on error in `exporters/raw.py`, `exporters/xml.py`, `exporters/json.py`, `exporters/templates.py`, `exporters/values.py`, `exporters/pretty.py`, `exporters/html.py`, `exporters/latex.py`, `metacommands/debug.py`, `metacommands/control.py`, and `metacommands/prompt.py`. Previously, an exception during export could leak open file handles.
+
+- XML export injection: `exporters/xml.py` now escapes `<`, `>`, `&` in cell values using `xml.sax.saxutils.escape()` and sanitizes `--` in XML comments. Previously, data containing XML special characters produced malformed XML output.
+
+- HTML export XSS: `exporters/html.py` now escapes column headers and cell values using `html.escape()`. Previously, data containing `<script>` or other HTML markup was written directly to the output file.
+
+- JSON export injection: `exporters/json.py` now uses `json.dumps()` to properly escape the `description` field in JSON-TS exports. Previously, descriptions containing quotes or backslashes produced malformed JSON. Also added missing `import json` to `write_query_to_json_ts` which previously relied on a global set by `write_query_to_json`.
+
+- `JinjaTemplateReport.__repr__` incorrectly returned `StrTemplateReport(...)` instead of `JinjaTemplateReport(...)`.
+
+- `exporters/templates.py`: removed bare `except: raise` clause in `JinjaTemplateReport.write_report()` and added proper exception chaining (`from e`) to Jinja2 template errors.
+
+- `exporters/delimited.py`: `write_delimited_file()` now closes the output file handle via `try/finally`. Previously, the file was opened but never closed, leaking the handle on both success and error paths.
+
+- SQL injection in remaining database metadata queries: `role_exists()` in `db/mysql.py`, `db/sqlserver.py`, and `db/firebird.py`, `schema_exists()` in `db/sqlserver.py`, `table_exists()` and `view_exists()` in `db/access.py` and `db/firebird.py` now use parameterized queries instead of f-string interpolation. `column_exists()` and `table_columns()` in `db/access.py` and `db/firebird.py` now use `quote_identifier()` for SQL identifiers that cannot be parameterized.
+
+- SQL injection in `db/postgres.py`: `create_db()` now uses `quote_identifier()` for database name and encoding in `CREATE DATABASE` DDL. COPY command delimiter and quote character are now escaped/validated to prevent injection.
+
+- CPU busy-loop in `utils/fileio.py`: `FileWriter.run()` now uses blocking `queue.get(timeout=0.1)` instead of `get_nowait()` in a tight loop, eliminating unnecessary CPU consumption when the file writer subprocess is idle.
+
+- Substitution variable cycle detection in `script.py`: `substitute_vars()` now enforces a maximum of 100 iterations to prevent infinite loops when variables reference each other cyclically (e.g., `$A` expands to `!!$B!!` and `$B` expands to `!!$A!!`).
+
+- Jinja2 template injection: `exporters/templates.py` now uses `jinja2.sandbox.SandboxedEnvironment` instead of the default `jinja2.Template` constructor. Previously, a malicious template file could access Python internals and execute arbitrary code.
+
+- SQL injection in `import_entire_file()` across 6 database backends (`db/base.py`, `db/dsn.py`, `db/sqlite.py`, `db/sqlserver.py`, `db/postgres.py`, `db/access.py`): the `column_name` parameter is now quoted with `quote_identifier()` instead of being interpolated directly into the INSERT statement.
+
+- XML export malformed output: `exporters/xml.py` now sanitizes column headers and table names used as XML element names, replacing invalid XML name characters with underscores. Previously, column names containing `>`, `<`, or other XML metacharacters produced malformed XML.
+
+- `$SHEETS_TABLES_VALUES` SQL injection in `metacommands/io_import.py`: sheet names from ODS/XLS imports are now escaped (single quotes doubled) before embedding in SQL value expressions. Previously, a sheet name containing a single quote produced malformed or injectable SQL.
+
+- HTTP header injection in SERVE metacommand (`metacommands/io_fileops.py`): the `Content-Disposition` filename is now sanitized (newlines/carriage returns stripped, quotes escaped) to prevent HTTP response splitting.
+
+- Empty `dt_cast` type-cast mapping in `Database` base class (`db/base.py`): the monolith populated this dict with 8 type converters (int, float, str, bool, datetime, date, Decimal, bytearray) but the refactored version was initialized to `{}`. Now uses a lazy property that auto-populates on first access, ensuring all backends get the correct type casters even though none call `super().__init__()`.
+
+- `WriteSpec.write()` file descriptor leak (`exporters/base.py`): the one-liner `EncodedFile(...).open("a").write(msg)` opened a file, wrote, and discarded the handle without closing. Now properly closes the handle in a try/finally block.
+
+- Removed duplicate `x_halt_msg` function from `metacommands/prompt.py`. The canonical version in `metacommands/control.py` (used by the dispatch table) was the correct one; the prompt.py copy was dead code.
+
+- `read_sqlfile()` double-open file leak (`script.py`): `ScriptFile.__init__` already opens the file, but `read_sqlfile()` called `.open("r")` again, creating a second leaked handle. Now uses the ScriptFile directly.
+
+- Missing WRITE metacommand delimiter patterns in dispatch table (`metacommands/__init__.py`): added 5 missing delimiter variants (tilde, hash, backtick, bracket, single-quote) for WRITE and ON ERROR_HALT/ON CANCEL_HALT WRITE metacommands that were present in the monolith but missing from the refactored dispatch table.
+
+- Missing bare (non-CONFIG) settings aliases in dispatch table (`metacommands/__init__.py`): added 10 bare settings forms (e.g., `FEEDBACK ON` without the `CONFIG` prefix) that the monolith supported but the refactored dispatch table omitted.
+
+- `CONNECT TO DSN` metacommand unreachable (`metacommands/__init__.py`): the `x_connect_dsn` handler was imported but never registered with `mcl.add()`. DSN-based connections via metacommand were completely broken.
+
+- `PARQUET` missing from EXPORT format regex lists (`metacommands/__init__.py`): the export handler code supported Parquet output but the dispatch table regex didn't include `PARQUET` as a valid format, making `EXPORT ... AS PARQUET` unreachable. Added to both EXPORT QUERY and EXPORT table format lists.
+
+- Firebird error message typo (`db/firebird.py`): error message said "required to connect to MySQL" instead of "required to connect to Firebird".
+
+- Oracle default port in function signature (`db/oracle.py`): default port parameter was `5432` (PostgreSQL's port) instead of `1521` (Oracle's port). The body already corrected to 1521 but the signature was misleading.
+
+- Missing documentation for `show_progress` and `log_sql` config options in `docs/configuration.md`.
+
+- Backward compatibility for `TXT-AND` export format (`metacommands/__init__.py`, `metacommands/io_export.py`): the monolith used `TXT-AND`/`TEXT-AND` but the refactored code renamed it to `TXT-AND`/`TEXT-AND`. Added `TXT-AND` as a backward-compatible alias so existing scripts continue to work.
+
+- `docs/api/db.md` missing individual database adapter documentation — added all 9 adapters (postgres, sqlite, duckdb, sqlserver, mysql, oracle, firebird, access, dsn).
+
+- `docs/api/exporters.md` missing `latex` and `zip` module references — added both.
+
+______________________________________________________________________
+
+## [2.1.2] - 2026-03-25
+
+### Added
+
+- DuckDB integration tests: 15 end-to-end tests (`tests/test_integration_duckdb.py`) covering basic SQL, substitution variables, CSV export/import, conditional execution, WRITE metacommand, round-trip, and DuckDB-specific features (views, schemas, native types).
+
+### Changed
+
+- Added PyPI version, Python versions, license, and Read the Docs badges to `README.md`.
+
+### Fixed
+
+- Config parser now accepts `k` (DuckDB) as a valid `db_type` in `execsql.conf`. Previously only the CLI flag `-t k` worked; config file validation rejected it.
+
+- Read the Docs build: added `mkdocstrings-python` and editable project install to `.readthedocs.yaml` so `mkdocstrings` can resolve API references.
+
+- Fixed escaped underscore in `docs/api/cli.md` (`\_run` → `_run`) that caused `mkdocstrings` to fail resolving `execsql.cli._run`.
+
+- Excluded `docs/api/` from `mdformat` pre-commit hook to prevent it from re-escaping underscores in `mkdocstrings` `:::` directives.
+
+______________________________________________________________________
+
 ## [2.1.1] - 2026-03-25
 
 ### Added

{execsql2-2.1.1 → execsql2-2.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: execsql2
-Version: 2.1.1
+Version: 2.2.1
 Summary: Runs a SQL script against a PostgreSQL, SQLite, MariaDB/MySQL, DuckDB, Firebird, MS-Access, MS-SQL-Server, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables.
 Project-URL: Repository, https://github.com/geocoug/execsql
 Project-URL: Issues, https://github.com/geocoug/execsql/issues
@@ -98,14 +98,15 @@ Provides-Extra: postgres
 Requires-Dist: psycopg2-binary; extra == 'postgres'
 Description-Content-Type: text/markdown
 
-> [!WARNING]
-> **This project is in active development and is not yet stable.**
-> The codebase is being modernized from the upstream monolith into a modular package.
-> APIs, CLI behavior, and configuration options may change without notice.
-> Not recommended for production use. For the stable upstream release, see [execsql](https://execsql.readthedocs.io/).
+> [!NOTE]
+> **This is a maintained fork of [execsql](https://execsql.readthedocs.io/).**
+> The original monolith has been fully refactored into a modular package with 2,000+ tests.
+> The CLI and configuration are backwards-compatible with upstream v1.130.1.
+> Report issues at [github.com/geocoug/execsql/issues](https://github.com/geocoug/execsql/issues).
 
 <div align="center">
-![execsql logo](https://execsql2.readthedocs.io/en/latest/images/execsql_logo_01.png)
+
+<img src="https://execsql2.readthedocs.io/en/latest/images/execsql_logo_01.png" alt="execsql logo">
 
 *Multi-DBMS SQL script processor.*
 
@@ -115,6 +116,10 @@ Description-Content-Type: text/markdown
 
 [![CI/CD](https://github.com/geocoug/execsql/actions/workflows/ci-cd.yml/badge.svg)](https://github.com/geocoug/execsql/actions/workflows/ci-cd.yml)
 [![codecov](https://codecov.io/gh/geocoug/execsql/graph/badge.svg)](https://codecov.io/gh/geocoug/execsql)
+[![Docs](https://readthedocs.org/projects/execsql2/badge/)](https://execsql2.readthedocs.io/)
+[![PyPI](https://img.shields.io/pypi/v/execsql2)](https://pypi.org/project/execsql2/)
+[![Python](https://img.shields.io/pypi/pyversions/execsql2)](https://pypi.org/project/execsql2/)
+[![License](https://img.shields.io/pypi/l/execsql2)](https://pypi.org/project/execsql2/)
 [![Downloads](https://pepy.tech/badge/execsql2/month)](https://pepy.tech/project/execsql2)
 
 </div>

{execsql2-2.1.1 → execsql2-2.2.1}/README.md

@@ -1,11 +1,12 @@
-> [!WARNING]
-> **This project is in active development and is not yet stable.**
-> The codebase is being modernized from the upstream monolith into a modular package.
-> APIs, CLI behavior, and configuration options may change without notice.
-> Not recommended for production use. For the stable upstream release, see [execsql](https://execsql.readthedocs.io/).
+> [!NOTE]
+> **This is a maintained fork of [execsql](https://execsql.readthedocs.io/).**
+> The original monolith has been fully refactored into a modular package with 2,000+ tests.
+> The CLI and configuration are backwards-compatible with upstream v1.130.1.
+> Report issues at [github.com/geocoug/execsql/issues](https://github.com/geocoug/execsql/issues).
 
 <div align="center">
-![execsql logo](https://execsql2.readthedocs.io/en/latest/images/execsql_logo_01.png)
+
+<img src="https://execsql2.readthedocs.io/en/latest/images/execsql_logo_01.png" alt="execsql logo">
 
 *Multi-DBMS SQL script processor.*
 
@@ -15,6 +16,10 @@
 
 [![CI/CD](https://github.com/geocoug/execsql/actions/workflows/ci-cd.yml/badge.svg)](https://github.com/geocoug/execsql/actions/workflows/ci-cd.yml)
 [![codecov](https://codecov.io/gh/geocoug/execsql/graph/badge.svg)](https://codecov.io/gh/geocoug/execsql)
+[![Docs](https://readthedocs.org/projects/execsql2/badge/)](https://execsql2.readthedocs.io/)
+[![PyPI](https://img.shields.io/pypi/v/execsql2)](https://pypi.org/project/execsql2/)
+[![Python](https://img.shields.io/pypi/pyversions/execsql2)](https://pypi.org/project/execsql2/)
+[![License](https://img.shields.io/pypi/l/execsql2)](https://pypi.org/project/execsql2/)
 [![Downloads](https://pepy.tech/badge/execsql2/month)](https://pepy.tech/project/execsql2)
 
 </div>

{execsql2-2.1.1 → execsql2-2.2.1}/docs/api/cli.md

@@ -16,6 +16,6 @@ members: false
 
 ## Core execution
 
-::: execsql.cli.\_run
+::: execsql.cli._run
 options:
 show_signature_annotations: false

{execsql2-2.1.1 → execsql2-2.2.1}/docs/api/db.md

@@ -13,3 +13,23 @@ If you are adding support for a new database, start with the [Adding Database Ad
 Convenience constructors used internally to create typed `Database` instances. The CLI calls the appropriate factory function based on the `-t` flag value. Each factory validates its arguments (e.g., checking that a file exists) before constructing and returning the adapter.
 
 ::: execsql.db.factory
+
+## Adapters
+
+::: execsql.db.postgres
+
+::: execsql.db.sqlite
+
+::: execsql.db.duckdb
+
+::: execsql.db.sqlserver
+
+::: execsql.db.mysql
+
+::: execsql.db.oracle
+
+::: execsql.db.firebird
+
+::: execsql.db.access
+
+::: execsql.db.dsn

{execsql2-2.1.1 → execsql2-2.2.1}/docs/api/exporters.md

@@ -26,6 +26,8 @@ Each module below implements one or more output formats. Every writer follows th
 
 ::: execsql.exporters.templates
 
+::: execsql.exporters.latex
+
 ::: execsql.exporters.ods
 
 ::: execsql.exporters.xls
@@ -37,3 +39,5 @@ Each module below implements one or more output formats. Every writer follows th
 ::: execsql.exporters.sqlite
 
 ::: execsql.exporters.duckdb
+
+::: execsql.exporters.zip

{execsql2-2.1.1 → execsql2-2.2.1}/docs/configuration.md

@@ -152,6 +152,10 @@ The section and property names that may be used in a configuration file are list
 
 :   Replaces newline characters that are in text values on [IMPORT](metacommands.md#import). Every sequence of a newline and any surrounding whitespace is replaced with a single space character.
 
+`show_progress`
+
+:   Whether or not to display a rich progress bar during long-running IMPORT operations. The property value should be either "Yes" or "No". The default is "No". This can also be enabled via the `--progress` CLI flag or the `CONFIG SHOW_PROGRESS` metacommand. Requires the `rich` Python package.
+
 `scan_lines` { #scan_lines }
 
 :   The number of lines of a data file to scan to determine the quoting character and delimiter character used. This is equivalent to the `-s` command-line option. The default value is 100.
@@ -306,6 +310,9 @@ The section and property names that may be used in a configuration file are list
 `linux_config_file`
 :   The full name or path to an additional configuration file to be read if *execsql* is running on Linux. If only a path is specified, the name of the configuration file should be `execsql.conf`. The configuration file specified will be read immediately following the configuration file in which it is named. No configuration file will be read more than once. If the name or path are invalid, this setting will be silently ignored. This setting may include [substitution variables](substitution_vars.md#substitution_vars); at the time that configuration files are read, however, only environment variables and system variables related to the script name and path are defined.
 
+`log_sql` { #log_sql }
+:   When set to "Yes", all executed SQL statements are written to the log file with a `sql` record type, including the database name, line number, and query text. The property value should be either "Yes" or "No". The default is "No". This can also be enabled via the `CONFIG LOG_SQL` metacommand.
+
 `max_log_size_mb` { #max_log_size_mb }
 :   Maximum size of the log file in megabytes before it is rotated. When set to a positive integer, the log file is rotated to `.1` before a new run appends to it if the file size exceeds the configured threshold. The default is `0` (disabled — no rotation).
 

{execsql2-2.1.1 → execsql2-2.2.1}/docs/dev/adding_metacommands.md

@@ -89,19 +89,22 @@ Open `src/execsql/metacommands/__init__.py` and add a `mcl.add()` call inside `b
 mcl.add(
     r"^\s*MY_COMMAND\s+(?P<target>\w+)\s+(?P<value>.+)$",
     x_my_command,
+    description="MY_COMMAND",
+    category="action",
 )
 ```
 
 `MetaCommandList.add()` accepts these parameters:
 
-| Parameter          | Type                       | Default  | Purpose                                                                                |
-| ------------------ | -------------------------- | -------- | -------------------------------------------------------------------------------------- |
-| `matching_regexes` | `str` or `tuple[str, ...]` | required | One regex string, or a tuple of strings all mapped to the same handler                 |
-| `exec_func`        | callable                   | required | The handler function                                                                   |
-| `description`      | `str \| None`              | `None`   | Human-readable label (shown in `DEBUG WRITE METACOMMANDLIST`)                          |
-| `run_in_batch`     | `bool`                     | `False`  | Allow execution inside an open `BEGIN_BATCH`/`END_BATCH` block                         |
-| `run_when_false`   | `bool`                     | `False`  | Execute even when the `IF`-stack condition is false (needed for `ELSE`, `ENDIF`, etc.) |
-| `set_error_flag`   | `bool`                     | `True`   | Update `_state.status.metacommand_error` on success/failure                            |
+| Parameter          | Type                       | Default  | Purpose                                                                                                           |
+| ------------------ | -------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------- |
+| `matching_regexes` | `str` or `tuple[str, ...]` | required | One regex string, or a tuple of strings all mapped to the same handler                                            |
+| `exec_func`        | callable                   | required | The handler function                                                                                              |
+| `description`      | `str \| None`              | `None`   | Human-readable keyword name (shown in `DEBUG WRITE METACOMMANDLIST` and `--dump-keywords`)                        |
+| `run_in_batch`     | `bool`                     | `False`  | Allow execution inside an open `BEGIN_BATCH`/`END_BATCH` block                                                    |
+| `run_when_false`   | `bool`                     | `False`  | Execute even when the `IF`-stack condition is false (needed for `ELSE`, `ENDIF`, etc.)                            |
+| `set_error_flag`   | `bool`                     | `True`   | Update `_state.status.metacommand_error` on success/failure                                                       |
+| `category`         | `str \| None`              | `None`   | Keyword category for `--dump-keywords` and VS Code grammar generation (e.g., `"action"`, `"control"`, `"config"`) |
 
 All regexes are compiled with `re.I` (case-insensitive) automatically.
 
@@ -209,9 +212,10 @@ ______________________________________________________________________
 
 - [ ] Handler function added to the appropriate `src/execsql/metacommands/*.py` module
 - [ ] Handler imported in `src/execsql/metacommands/__init__.py`
-- [ ] `mcl.add(...)` call added in `build_dispatch_table()` at the right position
+- [ ] `mcl.add(...)` call added in `build_dispatch_table()` with `description=` and `category=`
+- [ ] Run `just generate-vscode-grammar` to update the VS Code grammar
 - [ ] Integration test added to `tests/test_metacommands.py` (or relevant file)
-- [ ] `pytest` passes locally
+- [ ] `pytest` passes locally (including `tests/test_registry.py` keyword consistency checks)
 
 ______________________________________________________________________
 

{execsql2-2.1.1 → execsql2-2.2.1}/docs/index.md

@@ -8,17 +8,17 @@
 
 You can use *execsql* to:
 
-> - Import data from text files and spreadsheets into a database.
-> - Export tables and views to text files, OpenDocument spreadsheets, HTML, JSON, LaTeΧ, XML, or 17 other data formats.
-> - Copy data between different databases, even databases using different DBMSs.
-> - Display tables or views on the terminal or in a GUI dialog window.
-> - Export data using template processors to produce non-tabular output with customized format and contents.
-> - Conditionally execute different SQL commands and metacommands based on the DBMS in use, the database in use, data values, user input, and other conditions.
-> - Execute blocks of SQL statements and metacommands repeatedly, using any of three different looping methods.
-> - Prompt the user to select files or directories, answer questions, or enter data values.
-> - Allow the user to visually compare two tables or views.
-> - Serve data tables to a web application when used as a CGI script.
-> - Write messages to the console or to a file during the processing of a SQL script. These messages can be used to display the progress of the script or create a custom log of the operations that have been carried out or results obtained. Status messages and data exported in text format can be combined in a single text file. Data tables can be exported in a text format that is compatible with Markdown pipe tables, so that script output can be converted into a variety of document formats (see [Example 8](examples.md#example8) and [Example 11](examples.md#example11)).
+- Import data from text files and spreadsheets into a database.
+- Export tables and views to text files, OpenDocument spreadsheets, HTML, JSON, LaTeΧ, XML, or 17 other data formats.
+- Copy data between different databases, even databases using different DBMSs.
+- Display tables or views on the terminal or in a GUI dialog window.
+- Export data using template processors to produce non-tabular output with customized format and contents.
+- Conditionally execute different SQL commands and metacommands based on the DBMS in use, the database in use, data values, user input, and other conditions.
+- Execute blocks of SQL statements and metacommands repeatedly, using any of three different looping methods.
+- Prompt the user to select files or directories, answer questions, or enter data values.
+- Allow the user to visually compare two tables or views.
+- Serve data tables to a web application when used as a CGI script.
+- Write messages to the console or to a file during the processing of a SQL script. These messages can be used to display the progress of the script or create a custom log of the operations that have been carried out or results obtained. Status messages and data exported in text format can be combined in a single text file. Data tables can be exported in a text format that is compatible with Markdown pipe tables, so that script output can be converted into a variety of document formats (see [Example 8](examples.md#example8) and [Example 11](examples.md#example11)).
 
 Different DBMSs and DBMS-specific client programs provide different and incompatible extensions to SQL, ordinarily to allow interactions with the file system and to allow conditional tests and looping. Some DBMSs do not have any native extensions of this sort. *execsql* provides these features, as well as features for user interaction, in an identical fashion for all supported DBMSs. This allows standardization of the SQL scripting language used for different types of database management systems.