sqlnbfmt 0.5.0__tar.gz → 0.6.0__tar.gz

{sqlnbfmt-0.5.0 → sqlnbfmt-0.6.0}/.github/workflows/ci-and-publish.yml

@@ -29,6 +29,8 @@ jobs:
       - run: uv python install 3.12
       - run: uv sync --all-extras --dev --frozen
       - run: uv run pytest
+      # Also the oldest sqlglot that pyproject.toml allows (keep in sync)
+      - run: uv run --isolated --with "sqlglot==25.27.0" pytest
 
   security:
     runs-on: ubuntu-latest

{sqlnbfmt-0.5.0 → sqlnbfmt-0.6.0}/CHANGELOG.md

@@ -5,6 +5,42 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.6.0] - 2026-10-16
+
+**Upgrading is recommended.** Earlier versions could change what code means: query parameters were rewritten with the postgres, duckdb and bigquery dialects, non-ASCII text could cut a rewrite in the wrong place, f-string fields lost their quotes, and `%sql` line magics deleted the rest of the cell. sqlnbfmt now checks every rewrite and leaves a string unchanged, with a warning, when it can't format it safely.
+
+### Added
+
+- `extend_sql_keywords`, `extend_function_names` and `extend_sql_decorators` config options, which add entries to the built-in lists instead of replacing them, so adding one function name no longer drops defaults like `sql` (Marimo's `mo.sql()`) ([#26](https://github.com/flyersworder/sqlnbfmt/issues/26))
+- The unknown-option warning suggests the intended option for kebab-case keys (e.g. `extend-function-names`, `indent-width`), at the top level and under `formatting_options`
+
+### Deprecated
+
+- `formatting_options.single_line_threshold`: it was documented as the maximum length before splitting SQL, but no version ever read it; line breaks are decided by sqlglot. Setting it now logs a warning, and it will be removed in a future release
+
+### Fixed
+
+- **Query parameters were corrupted with the postgres, duckdb and bigquery dialects.** `%s`/`?` parameters became `%(AUTO_PLACEHOLDER_0)s`, `$AUTO_PLACEHOLDER_0` or `@AUTO_PLACEHOLDER_0`, f-string fields became `%({table})s`, `${table}` or `@{table}`, and user-written `:name` parameters were converted to the dialect's own style. All parameter styles (`%s`, `?`, `%(name)s`, `:name`, `:1`, `$1`) and f-string fields now pass through unchanged; `?` or `%s` inside SQL string literals and comments is left as text, using sqlglot's tokenizer for the dialect to tell them apart (so `'\'` in postgres, `$$…$$` bodies and MySQL/BigQuery `#` comments are read correctly). Queries with 11 or more parameters were also mangled (`_1` matched inside `_10`)
+- **Formatting could move parameters or rewrap fields.** Positional parameters bind by order, but `… OFFSET %s LIMIT %s` was reordered to `LIMIT %s OFFSET %s` (and T-SQL `LIMIT ?` to `TOP (?)`), a field holding a clause, as in `f"SELECT * FROM t {where_clause}"`, became `FROM t AS {where_clause}`, and `SELECT {distinct} a` became `SELECT {distinct} AS a`. Parameters and fields could also be pulled into a string (`INTERVAL %s DAY` → `INTERVAL '%s DAY'`) or wrapped in parentheses (`VALUES %s` → `VALUES (%s)`, which breaks psycopg2's `execute_values`), and the quotes around a field could be rewritten (`"{name}"` → `'{name}'` in MySQL, `$${body}$$` → `'{body}'`), which breaks when the value contains a quote. A `%s` inside a SQL comment, which drivers fill in too, could be moved or dropped along with the comment, shifting the values of the parameters after it. Positional parameters must now keep their order (comments included), fields and parameters must stay outside strings without added parentheses, a string holding a field must be written exactly as before, and one may only become an alias, or be given one, if the original wrote `AS` there; otherwise the string is left unformatted with a warning that says why. Harmless rewrites (comments, a trailing `;`, `%s::int` → `CAST(%s AS INT)`) are still formatted
+- **Multi-line SQL string literals changed when re-indented.** In SQL passed to a call, the continuation lines of a string such as `'multi\nline'`, a quoted name (`` `x\ny` ``, `[x\ny]`) or a `$$…$$` function body were indented too, changing its value; only lines outside SQL strings are indented now, following the dialect's quoting rules
+- **Magic cells lost code.** A `%sql` line magic deleted every line after it, `%%sql` arguments were dropped (`%%sql df <<` lost the assignment, `%%sql postgresql://…` the connection), a `%%sql` cell with lone `\r` line breaks lost its SQL, and a line magic collapsed spaces inside strings (`'a  b'` → `'a b'`). Line breaks around the magic are now kept as written
+- **Multi-line block comments never settled.** In SQL passed to a call, `/* … */` comments spanning lines were indented further on every run, so `--check` always failed
+- `%%` (an escaped `%` in pyformat SQL) followed by `s` was read as a `%s` parameter
+- **Non-ASCII text could break the file.** AST column offsets count UTF-8 bytes, but they were used as character positions, so with `é`, `日本語` or emoji on the same line, a rewrite was cut at the wrong place (e.g. deleting the closing `)` of `cursor.execute("… 'é' …")`). A lone `\r` line break could crash the run; line breaks are now counted like Python does, and an unexpected error skips the cell or file with a warning instead of aborting
+- **F-string fields lost their quotes or format.** `b = '{y}'` became `b = {y}`, conversions and format specs (`{x!r}`, `{y:>3}`) were dropped, and field expressions were rewritten (`{d["k"]}` became `{d['k']}`). Fields are now copied from the source verbatim. F-strings containing backslashes or literal braces are now formatted correctly instead of being skipped
+- **Re-parse check:** formatted SQL is parsed again and must give the same syntax tree as the original (ignoring quoting of names), so a printing bug in sqlglot can't change a query. Before sqlglot 30.23, for example, the second line of a multi-line quoted name was indented
+- **Whole-file safety net:** after all rewrites, the edited source is parsed again and every string literal is compared with the original (untouched ones) or its intended value (rewritten ones). A rewrite that fails the check is dropped and reported with its line number (the others are still applied), with a request for a bug report. Strings that are skipped because they can't be written back safely are now reported as warnings with their line number
+- **`sql_decorators` now works.** The option was parsed and documented but never used by the formatter in any version ([#30](https://github.com/flyersworder/sqlnbfmt/issues/30)). SQL in decorator arguments (`@query("SELECT ...")`) and string literals returned from decorated functions (`@query` above `def f(): return "SELECT ..."`) are now formatted. Decorator names match like function names, and docstrings and returns of nested functions are left alone
+- **Rewritten SQL strings could corrupt the code.** When writing formatted SQL back, sqlnbfmt now keeps the value of the string exactly. In particular:
+  - backslash escapes are preserved (`'\\n'` used to become a real newline)
+  - the raw `r` prefix is kept (dropping it turned `\_` into an invalid escape)
+  - quotes are chosen so they don't clash with the SQL (`cursor.execute('delete from "users"')` produced broken Python, as did `"""` inside the SQL)
+  - literal `{{ }}` braces in f-strings are no longer turned into placeholders
+
+  Each rewritten literal is parsed back and checked; a string that can't be written back faithfully is left unchanged
+- `sql_keywords` entries are now case-insensitive. SQL detection compares against upper-cased code, so a lowercase entry such as `"merge"` was silently never matched
+- Blank entries in `sql_keywords`, `function_names` and `sql_decorators` (and their `extend_*` forms) are rejected. Function names match as substrings, so `""` matched every call
+
 ## [0.5.0] - 2026-10-15
 
 ### Added

{sqlnbfmt-0.5.0 → sqlnbfmt-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: sqlnbfmt
-Version: 0.5.0
+Version: 0.6.0
 Summary: A tool to format SQL code in Jupyter notebooks and Marimo notebooks.
 Project-URL: Homepage, https://github.com/flyersworder/sqlnbfmt
 Project-URL: Repository, https://github.com/flyersworder/sqlnbfmt
@@ -121,7 +121,7 @@ pip install pre-commit
 ```yaml
 repos:
   - repo: https://github.com/flyersworder/sqlnbfmt
-    rev: v0.5.0
+    rev: v0.6.0
     hooks:
       - id: sqlnbfmt
         types: [jupyter]
@@ -169,6 +169,7 @@ Use `--check` in GitHub Actions to enforce formatting:
 ```toml
 [tool.sqlnbfmt]
 dialect = "postgres"
+extend_function_names = ["run_query"]  # recognized in addition to the defaults
 
 [tool.sqlnbfmt.formatting_options]
 indent_width = 2
@@ -188,12 +189,25 @@ When `--config` is given, `pyproject.toml` discovery is skipped. Options you don
 |--------|-------------|---------|
 | `dialect` | SQL dialect passed to sqlglot (e.g. `postgres`, `mysql`, `bigquery`); `--dialect` overrides it | none (generic SQL) |
 | `sql_keywords` | SQL keywords to recognize and format | Common SQL keywords |
-| `function_names` | Python functions containing SQL code | `read_sql`, `execute`, `sql` (Marimo's `mo.sql`), etc. |
-| `sql_decorators` | Decorators indicating SQL code | `query`, `sql_query`, etc. |
-| `formatting_options.single_line_threshold` | Maximum length before splitting SQL | 80 |
+| `function_names` | Python functions containing SQL code (see [matching](#how-function-and-decorator-names-match)) | `read_sql`, `execute`, `sql` (Marimo's `mo.sql`), etc. |
+| `sql_decorators` | Decorators whose SQL is formatted: string arguments of `@query("SELECT ...")`, and strings returned by a function decorated with `@query` | `query`, `db_query`, `sql_query`, `sql_decorator` |
+| `extend_sql_keywords`, `extend_function_names`, `extend_sql_decorators` | Entries added to the corresponding list instead of replacing it | none |
+| `formatting_options.single_line_threshold` | **Deprecated:** never had any effect and will be removed. Line breaks are decided by sqlglot | |
 | `formatting_options.indent_width` | Number of spaces for indentation | 4 |
 
-`sql_keywords`, `function_names` and `sql_decorators` **replace** the built-in lists rather than extending them. Copy the defaults you still rely on from the [template](https://github.com/flyersworder/sqlnbfmt/blob/main/config.yaml); for example, keep `sql` in `function_names` or Marimo's `mo.sql()` calls stop being formatted.
+To add to the built-in lists, use the `extend_*` options. `sql_keywords`, `function_names` and `sql_decorators` **replace** the built-in lists, so setting `function_names = ["run_query"]` drops `sql`, and Marimo's `mo.sql()` calls stop being formatted. If you do replace a list, copy the defaults you still rely on from the [template](https://github.com/flyersworder/sqlnbfmt/blob/main/config.yaml). You can set both: the `extend_*` entries are added to your replacement list.
+
+### How function and decorator names match
+
+A call counts as SQL-bearing when a configured name appears **anywhere** in its full dotted name. Matching is case-sensitive. This is deliberately permissive, so that wrappers and variants are covered without configuration:
+
+- `execute` matches `cursor.execute`, `cursor.executemany` and `execute_sql`
+- `sql` matches `mo.sql`, `spark.sql`, pandasql's `sqldf`/`pysqldf` and `sqlalchemy.text`
+- `sql` does **not** match `runSQLQuery`, because matching is case-sensitive; add `SQL` if you need it
+
+Decorator names in `sql_decorators` match the same way, against the decorator's dotted name (`query` matches `@query`, `@db_query` and `@app.query(...)`).
+
+A matched call's string arguments are only reformatted if they look like SQL. Still, prefer specific names: `run` would also match `rerun(...)` and `subprocess.run(...)`, so use `run_query` rather than `run`.
 
 ## Examples
 
@@ -255,7 +269,7 @@ def _(mo):
 
 **SQL not being formatted?**
 - Ensure the string contains at least 2 SQL keywords or a recognizable pattern like `SELECT...FROM`
-- Check that the function name is in the recognized list (add custom ones via `function_names` in `[tool.sqlnbfmt]` or `--config`)
+- Check that the function name is in the recognized list (add custom ones via `extend_function_names` in `[tool.sqlnbfmt]` or your `--config` file)
 
 **Comments being modified?**
 - Python comments (`#`) are preserved. SQL comments (`--`) inside strings are converted to `/* */` block comments by sqlglot.
@@ -283,6 +297,11 @@ source .venv/bin/activate  # On Windows: .venv\Scripts\activate
 3. Run tests:
 ```bash
 pytest
+```
+
+   Before a release, also run the live end-to-end tests. They build the wheel, install it into a fresh virtualenv with Jupyter Server and jupyterlab-code-formatter, and exercise the CLI, the pre-commit hooks and the Jupyter extension against a scratch project. They need `uv`, `git` and network access, so the default `pytest` run skips them:
+```bash
+uv run pytest -m live
 ```
 
 4. Add eval cases: see `tests/eval/generate_fixtures.py` for examples. Run `python tests/eval/generate_fixtures.py` to regenerate fixtures.

{sqlnbfmt-0.5.0 → sqlnbfmt-0.6.0}/README.md

@@ -93,7 +93,7 @@ pip install pre-commit
 ```yaml
 repos:
   - repo: https://github.com/flyersworder/sqlnbfmt
-    rev: v0.5.0
+    rev: v0.6.0
     hooks:
       - id: sqlnbfmt
         types: [jupyter]
@@ -141,6 +141,7 @@ Use `--check` in GitHub Actions to enforce formatting:
 ```toml
 [tool.sqlnbfmt]
 dialect = "postgres"
+extend_function_names = ["run_query"]  # recognized in addition to the defaults
 
 [tool.sqlnbfmt.formatting_options]
 indent_width = 2
@@ -160,12 +161,25 @@ When `--config` is given, `pyproject.toml` discovery is skipped. Options you don
 |--------|-------------|---------|
 | `dialect` | SQL dialect passed to sqlglot (e.g. `postgres`, `mysql`, `bigquery`); `--dialect` overrides it | none (generic SQL) |
 | `sql_keywords` | SQL keywords to recognize and format | Common SQL keywords |
-| `function_names` | Python functions containing SQL code | `read_sql`, `execute`, `sql` (Marimo's `mo.sql`), etc. |
-| `sql_decorators` | Decorators indicating SQL code | `query`, `sql_query`, etc. |
-| `formatting_options.single_line_threshold` | Maximum length before splitting SQL | 80 |
+| `function_names` | Python functions containing SQL code (see [matching](#how-function-and-decorator-names-match)) | `read_sql`, `execute`, `sql` (Marimo's `mo.sql`), etc. |
+| `sql_decorators` | Decorators whose SQL is formatted: string arguments of `@query("SELECT ...")`, and strings returned by a function decorated with `@query` | `query`, `db_query`, `sql_query`, `sql_decorator` |
+| `extend_sql_keywords`, `extend_function_names`, `extend_sql_decorators` | Entries added to the corresponding list instead of replacing it | none |
+| `formatting_options.single_line_threshold` | **Deprecated:** never had any effect and will be removed. Line breaks are decided by sqlglot | |
 | `formatting_options.indent_width` | Number of spaces for indentation | 4 |
 
-`sql_keywords`, `function_names` and `sql_decorators` **replace** the built-in lists rather than extending them. Copy the defaults you still rely on from the [template](https://github.com/flyersworder/sqlnbfmt/blob/main/config.yaml); for example, keep `sql` in `function_names` or Marimo's `mo.sql()` calls stop being formatted.
+To add to the built-in lists, use the `extend_*` options. `sql_keywords`, `function_names` and `sql_decorators` **replace** the built-in lists, so setting `function_names = ["run_query"]` drops `sql`, and Marimo's `mo.sql()` calls stop being formatted. If you do replace a list, copy the defaults you still rely on from the [template](https://github.com/flyersworder/sqlnbfmt/blob/main/config.yaml). You can set both: the `extend_*` entries are added to your replacement list.
+
+### How function and decorator names match
+
+A call counts as SQL-bearing when a configured name appears **anywhere** in its full dotted name. Matching is case-sensitive. This is deliberately permissive, so that wrappers and variants are covered without configuration:
+
+- `execute` matches `cursor.execute`, `cursor.executemany` and `execute_sql`
+- `sql` matches `mo.sql`, `spark.sql`, pandasql's `sqldf`/`pysqldf` and `sqlalchemy.text`
+- `sql` does **not** match `runSQLQuery`, because matching is case-sensitive; add `SQL` if you need it
+
+Decorator names in `sql_decorators` match the same way, against the decorator's dotted name (`query` matches `@query`, `@db_query` and `@app.query(...)`).
+
+A matched call's string arguments are only reformatted if they look like SQL. Still, prefer specific names: `run` would also match `rerun(...)` and `subprocess.run(...)`, so use `run_query` rather than `run`.
 
 ## Examples
 
@@ -227,7 +241,7 @@ def _(mo):
 
 **SQL not being formatted?**
 - Ensure the string contains at least 2 SQL keywords or a recognizable pattern like `SELECT...FROM`
-- Check that the function name is in the recognized list (add custom ones via `function_names` in `[tool.sqlnbfmt]` or `--config`)
+- Check that the function name is in the recognized list (add custom ones via `extend_function_names` in `[tool.sqlnbfmt]` or your `--config` file)
 
 **Comments being modified?**
 - Python comments (`#`) are preserved. SQL comments (`--`) inside strings are converted to `/* */` block comments by sqlglot.
@@ -255,6 +269,11 @@ source .venv/bin/activate  # On Windows: .venv\Scripts\activate
 3. Run tests:
 ```bash
 pytest
+```
+
+   Before a release, also run the live end-to-end tests. They build the wheel, install it into a fresh virtualenv with Jupyter Server and jupyterlab-code-formatter, and exercise the CLI, the pre-commit hooks and the Jupyter extension against a scratch project. They need `uv`, `git` and network access, so the default `pytest` run skips them:
+```bash
+uv run pytest -m live
 ```
 
 4. Add eval cases: see `tests/eval/generate_fixtures.py` for examples. Run `python tests/eval/generate_fixtures.py` to regenerate fixtures.

{sqlnbfmt-0.5.0 → sqlnbfmt-0.6.0}/config.yaml

@@ -28,6 +28,11 @@ sql_keywords:
   - OFFSET
   - DISTINCT
 
+# To add to the defaults instead of replacing them, use extend_function_names
+# (and extend_sql_keywords / extend_sql_decorators), e.g.:
+# extend_function_names:
+#   - run_query
+
 function_names:
   - read_sql
   - read_sql_query
@@ -49,5 +54,4 @@ sql_decorators:
 
 # Formatting options
 formatting_options:
-  single_line_threshold: 80  # Maximum length for single-line SQL
   indent_width: 4           # Number of spaces for indentation

{sqlnbfmt-0.5.0 → sqlnbfmt-0.6.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "sqlnbfmt"
-version = "0.5.0"
+version = "0.6.0"
 description = "A tool to format SQL code in Jupyter notebooks and Marimo notebooks."
 authors = [
   { name = "Qing", email = "qingye779@gmail.com" }
@@ -49,8 +49,10 @@ build-backend = "hatchling.build"
 
 [tool.pytest.ini_options]
 markers = [
-    "integration: mark a test as an integration test."
+    "integration: mark a test as an integration test.",
+    "live: end-to-end tests against the built wheel; slow, need uv, git and network (run with `pytest -m live`)",
 ]
+addopts = "-m 'not live'"
 
 [tool.hatch.build.targets.wheel.shared-data]
 "jupyter-config/sqlnbfmt.json" = "etc/jupyter/jupyter_server_config.d/sqlnbfmt.json"