pytae 3.1.2__tar.gz → 3.2.0__tar.gz

pytae-3.2.0/.githooks/pre-commit

@@ -0,0 +1,4 @@
+#!/bin/sh
+# Run notebooks/*.ipynb before allowing a commit. Enable with:
+#   git config core.hooksPath .githooks
+python scripts/run_notebooks.py

{pytae-3.1.2 → pytae-3.2.0}/CHANGELOG.md

@@ -2,6 +2,17 @@
 
 All notable changes to this project are documented in this file.
 
+## [3.2.0] - 2026-09-13
+
+### Added
+- `-sql` flag: run a real SQL query against the current view at that point in the pipeline, via `duckdb` (optional dependency, install with `pip install pytae[sql]`). The view is queryable as table `df` only — no file-derived alias, since the file is already named on the command line. `table` is a reserved SQL keyword and is deliberately **not** registered as an alias.
+- `pytae[sql]` optional extra (`duckdb>=0.9`), needed for `-sql`.
+- `pytae[plot]` optional extra (`matplotlib`), needed for `Plotter`.
+- `scripts/run_notebooks.py` executes every notebook under `notebooks/` end-to-end and is wired up as a local pre-commit hook (`.githooks/pre-commit`, enable with `git config core.hooksPath .githooks`); deliberately **not** part of the `pytest` suite. `pytae[notebooks]` optional extra (`nbclient`, `nbformat`, `ipykernel`, `scipy`) provides what it needs to run.
+
+### Changed
+- Docs: `-rename` is now cross-linked from the `-sql` section as an alternative to escaping spaced column names, and surfaced in the Contents TOC (`Conversion — -convert / -rename`).
+
 ## [3.1.2]
 
 ### Added

{pytae-3.1.2 → pytae-3.2.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.5
 Name: pytae
-Version: 3.1.2
-Summary: A package to enhance common day to day tasks in the life of a data scientist
+Version: 3.2.0
+Summary: A package to enhance common tasks in data and plotting
 Project-URL: Homepage, https://github.com/maddytae/pytae
 Project-URL: Bug Tracker, https://github.com/maddytae/pytae/issues
 Project-URL: CLI Docs, https://github.com/maddytae/pytae/blob/master/docs/CLI.md
@@ -20,10 +20,18 @@ Requires-Python: >=3.10
 Requires-Dist: pandas>=2.0
 Requires-Dist: pyarrow>=15.0.0
 Provides-Extra: dev
+Requires-Dist: duckdb>=0.9; extra == 'dev'
 Requires-Dist: matplotlib; extra == 'dev'
 Requires-Dist: pytest>=8.0.0; extra == 'dev'
+Provides-Extra: notebooks
+Requires-Dist: ipykernel; extra == 'notebooks'
+Requires-Dist: nbclient; extra == 'notebooks'
+Requires-Dist: nbformat; extra == 'notebooks'
+Requires-Dist: scipy; extra == 'notebooks'
 Provides-Extra: plot
 Requires-Dist: matplotlib; extra == 'plot'
+Provides-Extra: sql
+Requires-Dist: duckdb>=0.9; extra == 'sql'
 Description-Content-Type: text/markdown
 
 # pytae
@@ -59,10 +67,22 @@ See [docs/LIBRARY.md](https://github.com/maddytae/pytae/blob/master/docs/LIBRARY
 ```bash
 pytae data.parquet -head
 pytae data.parquet -qry "'species': 'Adelie'" -select species,body_mass_g -convert -o subset.csv
+pytae data.parquet -sql "select species, avg(body_mass_g) from df group by species"
 ```
 
 See [docs/CLI.md](https://github.com/maddytae/pytae/blob/master/docs/CLI.md) for the full flag reference.
 
+## Development
+
+```bash
+pip install -e ".[dev,notebooks]"
+pytest                                        # unit tests
+git config core.hooksPath .githooks           # run notebooks/*.ipynb before each commit
+python scripts/run_notebooks.py               # or run the notebook check manually
+```
+
+Notebooks are intentionally not part of the `pytest` suite (they're slower and exercise plotting/IO end-to-end); the pre-commit hook is the enforcement point instead.
+
 ## License
 
 MIT — see [LICENSE](https://github.com/maddytae/pytae/blob/master/LICENSE).

{pytae-3.1.2 → pytae-3.2.0}/README.md

@@ -31,10 +31,22 @@ See [docs/LIBRARY.md](https://github.com/maddytae/pytae/blob/master/docs/LIBRARY
 ```bash
 pytae data.parquet -head
 pytae data.parquet -qry "'species': 'Adelie'" -select species,body_mass_g -convert -o subset.csv
+pytae data.parquet -sql "select species, avg(body_mass_g) from df group by species"
 ```
 
 See [docs/CLI.md](https://github.com/maddytae/pytae/blob/master/docs/CLI.md) for the full flag reference.
 
+## Development
+
+```bash
+pip install -e ".[dev,notebooks]"
+pytest                                        # unit tests
+git config core.hooksPath .githooks           # run notebooks/*.ipynb before each commit
+python scripts/run_notebooks.py               # or run the notebook check manually
+```
+
+Notebooks are intentionally not part of the `pytest` suite (they're slower and exercise plotting/IO end-to-end); the pre-commit hook is the enforcement point instead.
+
 ## License
 
 MIT — see [LICENSE](https://github.com/maddytae/pytae/blob/master/LICENSE).

{pytae-3.1.2 → pytae-3.2.0}/docs/CLI.md

@@ -8,6 +8,7 @@ Inspect and convert tabular files (`.parquet`, `.csv`, `.txt`, `.dat`, `.sas7bda
 - [Sample datasets](#sample-datasets)
 - [Column selection — `-select`](#select)
 - [Row filtering — `-qry` / `-query`](#filtering)
+- [SQL — `-sql`](#sql)
 - [Aggregation (auto group columns) — `-agg_df`](#agg-df)
 - [Aggregation (explicit group columns) — `-group_by` + `-agg`](#group-by-agg)
 - [Broadcast — `-group_x`](#group-x)
@@ -18,7 +19,7 @@ Inspect and convert tabular files (`.parquet`, `.csv`, `.txt`, `.dat`, `.sas7bda
 - [Missing values — `-handle_missing`](#handle-missing)
 - [Reshape — `-long` / `-wide`](#reshape)
 - [Cross-tabulation — `-crosstab`](#crosstab)
-- [Conversion — `-convert`](#convert)
+- [Conversion — `-convert` / `-rename`](#convert)
 - [Display extras](#display-extras)
 - [Pandas defaults vs pytae-specific](#pandas-vs-pytae)
   - [`-wide` vs `-crosstab`](#wide-vs-crosstab)
@@ -195,9 +196,44 @@ pytae penguins.parquet -qry "'species': 'Adelie'" -select species,body_mass_g -h
 
 ---
 
+<a id="sql"></a>
+## SQL — `-sql`
+
+`-sql` runs a real SQL query against the current view at this point in the pipeline, using [duckdb](https://duckdb.org/) (an optional dependency — install with `pip install pytae[sql]`). The view is queryable as table **`df`, and only `df`** — the file itself is already named on the command line (`pytae penguins.parquet ...`), so there's no separate file-derived alias to remember (and no ambiguity if you later pipe a differently-named file through the same command). `table` is also deliberately not registered: it's a reserved SQL keyword, so `select * from table` fails to parse unless quoted, which defeats the point of a short default name.
+
+Unlike `-qry`, this is **standard SQL**, not pytae's dict syntax — column names with spaces need **double** quotes (`"bill length mm"`), not single quotes. Single quotes are string literals in SQL, e.g. `'Adelie'`; using them around a column name either errors or silently compares against a constant string instead of the column.
+
+```bash
+pytae penguins.parquet -sql "select species, body_mass_g from df where body_mass_g > 3500"
+pytae penguins.parquet -sql "select species, avg(body_mass_g) as avg_mass from df group by species"
+
+# a column name with a space: double quotes, not single quotes
+pytae data.parquet -sql 'select species from df where "bill length mm" > 40'
+
+# chains like any other op — runs on the current view, replaces it
+pytae penguins.parquet -select species,island,body_mass_g -sql "select * from df where island = 'Dream'" -shape
+
+# a query that needs BOTH a double-quoted identifier (space in the column name)
+# and a single-quoted string literal — escape the identifier's inner double quotes
+pytae penguins.parquet -select species,island,body_mass_g -sql "select \"col a\" from df where island = 'Dream'" -shape
+```
+
+Mixing a spaced identifier and a string literal in one `-sql` value means the shell has to see both `"` and `'` — one of them needs escaping. Two ways to handle it:
+
+- **Escape in place** (above): wrap the whole `-sql` value in double quotes and escape the identifier's quotes (`\"col a\"`); the string literal's single quotes pass through untouched.
+- **Rename the column first, then no quoting needed.** `-rename` only applies at `-convert`/write time (see [Convert / rename](#convert)), not mid-pipeline, so this is a two-step workflow — convert once to a space-free schema, then run `-sql` against that file with plain identifiers:
+
+  ```bash
+  pytae penguins.parquet -convert -rename "col a:col_a" -o clean.parquet
+  pytae clean.parquet -sql "select col_a from df where island = 'Dream'"
+  ```
+
+---
+
 <a id="agg-df"></a>
 ## Aggregation (auto group columns) — `-agg_df`
 
+
 Groups by all **non-numeric** columns and aggregates the rest. `n` is group count.
 
 ```python
@@ -428,7 +464,7 @@ pytae penguins.parquet -crosstab "index='species',columns='sex'" -dropna false
 ---
 
 <a id="convert"></a>
-## Conversion — `-convert`
+## Conversion — `-convert` / `-rename`
 
 Output format is the `-o` extension. Omitting `-o` writes `.csv` next to the source. Cannot write `.sas7bdat`.
 
@@ -451,6 +487,7 @@ pytae data.txt -dlim "|" -convert -o data.csv
 pytae data.dat -convert -o data.csv                   # .dat defaults to '|' delimiter
 pytae 'data/*.parquet' -convert
 pytae penguins.parquet -convert -rename "old_name:new_name,another:clean"
+pytae penguins.parquet -convert -rename "old name:new_name,another:clean"      # spaces in a name are fine — only "," and ":" are delimiters
 pytae data.csv -encoding latin-1 -convert -o data.parquet
 ```
 
@@ -510,6 +547,7 @@ Some flags/keys are thin passthroughs to standard pandas methods and parameter n
 | Flag / key | pytae convention |
 |---|---|
 | `-qry` | dict-based filter syntax (`'col': ('>', 5)`) — pytae's own `qry()`, not a pandas method |
+| `-sql` | real SQL via duckdb (not pandas) — the current view is registered as table `df` only (no file-derived alias) |
 | `-select`'s `contains=` / `startswith=` / `endswith=` / `regex=` / `dtype=` / `exclude_dtype=` | pytae's own column-picking vocabulary; no equivalent shorthand in plain pandas |
 | `-agg_df` | auto-detects group columns (every non-numeric column becomes a group key) — pandas' `groupby()` always requires you to name them |
 | `-group_x` | broadcasts a group aggregate to every row via `group=`/`v=`/`a=` keys — wraps pandas `transform()` as a ready-made verb |
@@ -664,6 +702,7 @@ pytae 'folder/*.parquet' -convert
 | `-select SPEC` | Restrict columns at this point in the pipeline (union in one spec; repeat to filter remaining) |
 | `-qry CONDITIONS` | Filter rows at this point (`df.qry()`); surrounding `{}` optional |
 | `-query EXPR` | Filter rows at this point (`df.query()`) |
+| `-sql QUERY` | Run a SQL query at this point via duckdb; view is table `df` |
 | `-sort_by COLUMNS [asc\|desc]` | Sort rows (default: ascending) |
 
 **Aggregate & reshape**

{pytae-3.1.2 → pytae-3.2.0}/notebooks/agg_df.ipynb

@@ -23,7 +23,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 35,
+   "execution_count": 1,
    "id": "712c9de3",
    "metadata": {},
    "outputs": [],
@@ -49,7 +49,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 36,
+   "execution_count": 2,
    "id": "a96e3eaa",
    "metadata": {},
    "outputs": [
@@ -135,7 +135,7 @@
        "4     Gentoo     Biscoe  124       47.504878  5076.016260"
       ]
      },
-     "execution_count": 36,
+     "execution_count": 2,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -150,7 +150,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 37,
+   "execution_count": 3,
    "id": "e49c71f4",
    "metadata": {},
    "outputs": [
@@ -235,7 +235,7 @@
        "2                59.6         624350.0       5076.016260           6300.0  "
       ]
      },
-     "execution_count": 37,
+     "execution_count": 3,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -259,7 +259,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 38,
+   "execution_count": 4,
    "id": "8cf7855a",
    "metadata": {},
    "outputs": [
@@ -358,7 +358,7 @@
        "4  124  "
       ]
      },
-     "execution_count": 38,
+     "execution_count": 4,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -374,7 +374,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 39,
+   "execution_count": 5,
    "id": "e601735d",
    "metadata": {},
    "outputs": [
@@ -456,7 +456,7 @@
        "3     No  Dinner  106   20.095660  3.126887      9.0"
       ]
      },
-     "execution_count": 39,
+     "execution_count": 5,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -473,7 +473,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 40,
+   "execution_count": 6,
    "id": "72a605ef",
    "metadata": {},
    "outputs": [
@@ -553,7 +553,7 @@
        "4     Gentoo     Biscoe    124     624350.0"
       ]
      },
-     "execution_count": 40,
+     "execution_count": 6,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -569,7 +569,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 41,
+   "execution_count": 7,
    "id": "39967327",
    "metadata": {},
    "outputs": [
@@ -655,7 +655,7 @@
        "4     Gentoo     Biscoe    124      124  5076.016260"
       ]
      },
-     "execution_count": 41,
+     "execution_count": 7,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -670,7 +670,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 42,
+   "execution_count": 8,
    "id": "3e3c4f7f",
    "metadata": {},
    "outputs": [
@@ -775,7 +775,7 @@
        "4              5000.0       5076.016260  "
       ]
      },
-     "execution_count": 42,
+     "execution_count": 8,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -798,7 +798,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 43,
+   "execution_count": 9,
    "id": "d4035d91",
    "metadata": {},
    "outputs": [
@@ -911,7 +911,7 @@
        "7     Gentoo     NaN   5       45.625000  4587.500000"
       ]
      },
-     "execution_count": 43,
+     "execution_count": 9,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -926,7 +926,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 44,
+   "execution_count": 10,
    "id": "f136b83f",
    "metadata": {},
    "outputs": [
@@ -1021,7 +1021,7 @@
        "5     Gentoo    Male  61       49.473770  5484.836066"
       ]
      },
-     "execution_count": 44,
+     "execution_count": 10,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -1044,7 +1044,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 45,
+   "execution_count": 11,
    "id": "1c35b544",
    "metadata": {},
    "outputs": [
@@ -1164,7 +1164,7 @@
        "9     Gentoo     Biscoe    Male  61"
       ]
      },
-     "execution_count": 45,
+     "execution_count": 11,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -1180,7 +1180,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "py_env (3.14.0.final.0)",
    "language": "python",
    "name": "python3"
   },
@@ -1194,7 +1194,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.6"
+   "version": "3.14.0"
   }
  },
  "nbformat": 4,