PySpark-Column-Selectors 0.1.3__py3-none-any.whl

pyspark_column_selectors-0.1.3.dist-info/METADATA

@@ -0,0 +1,674 @@
+Metadata-Version: 2.5
+Name: PySpark_Column_Selectors
+Version: 0.1.3
+Summary: Adds column selector functionality to pyspark
+Project-URL: bugs, https://github.com/tmichel3796/PySparkSelectors/issues
+Project-URL: changelog, https://github.com/tmichel3796/PySparkSelectors/releases
+Project-URL: documentation, https://tmichel3796.github.io/PySparkSelectors/
+Project-URL: homepage, https://github.com/tmichel3796/PySparkSelectors
+Author-email: "Trevor A. Michel" <Info@TrevorMichel.com>
+Maintainer-email: "Trevor A. Michel" <Info@TrevorMichel.com>
+License: MIT
+License-File: LICENSE
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: pyspark
+Requires-Dist: rich
+Requires-Dist: typer
+Requires-Dist: typing-extensions
+Description-Content-Type: text/markdown
+
+# PySpark Column Selectors
+
+A small object-oriented framework for selecting DataFrame columns by **dtype**, **position**, or **name pattern** instead of typing out column names by hand. A selector can be passed almost anywhere a column or list of columns is normally accepted, and selectors can be combined with set operations and chained with ordinary column transforms.
+
+![PyPI version](https://img.shields.io/pypi/v/PySpark_Column_Selectors.svg)
+
+Adds column selector functionality to pyspark
+
+-   GitHub: <https://github.com/tmichel3796/PySpark_Column_Selectors/>
+-   PyPI package: <https://pypi.org/project/PySpark_Column_Selectors/>
+-   Created by: [**Trevor A. Michel**](trevormichel.com) \| GitHub <https://github.com/tmichel3796> \| PyPI <https://pypi.org/user/tmichel3796/>
+-   Free software: MIT License
+
+## Features
+
+## Dependencies
+
+-   Python 3.8+
+-   PySpark 3.4 or newer (for full functionality)
+    -   PySpark 3.3 still works, except `T.TimestampNTZType` won't exist
+    -   `temporal()`/`datetime_()` fall back to matching only `T.TimestampType` on those older versions instead of raising.
+    -   `withColumns` with multiple selector-driven mutations in a single call (see [Using selectors with other DataFrame operations](#using-selectors-with-other-dataframe-operations)) requires PySpark 3.3+, since `DataFrame.withColumns` itself didn't exist before then.
+-   No third-party packages beyond PySpark itself -- everything else used (`functools`, `operator`, `re`, `typing`) is part of the Python standard library.
+-   Spark Connect (`pyspark.sql.connect`) is supported automatically when present, but is not required -- this framework works the same either way.
+
+## Importing
+
+``` python
+import PySparkSelectors as scs
+```
+
+Every example below assumes this import, and calls each selector as `scs.<function_name>(...)`.
+
+------------------------------------------------------------------------
+
+## Example data
+
+Every example in this document runs against the same small DataFrame:
+
+``` python
+df = spark.createDataFrame(
+    [
+        (1, "north", "east", 120.5, 88, True, "2024-01-15", 910, "2024-06-01 08:30:00"),
+        (2, "south", "west", 340.0, 95, False, "2024-03-22", 875, "2024-06-02 14:45:00"),
+    ],
+    ["id", "name", "region", "elevation", "totalscore", "isactive", "signupdate", "score2024", "lastseen"],
+)
+```
+
+| id | name | region | elevation | totalscore | isactive | signupdate | score2024 | lastseen |
+|--------|--------|--------|--------|--------|--------|--------|--------|--------|
+| 1 | north | east | 120.5 | 88 | true | 2024-01-15 | 910 | 2024-06-01 08:30:00 |
+| 2 | south | west | 340.0 | 95 | false | 2024-03-22 | 875 | 2024-06-02 14:45:00 |
+
+Dtypes: `id` (int), `name`/`region` (string), `elevation` (double), `totalscore`/`score2024` (int), `isactive` (boolean), `signupdate` (date), `lastseen` (timestamp).
+
+------------------------------------------------------------------------
+
+## Dtype-based selectors
+
+Dtype selectors match against the real `pyspark.sql.types` classes, not dtype strings. Import the types module alongside the selectors:
+
+``` python
+import pyspark.sql.types as T
+```
+
+### `by_dtype([...])`
+
+Matches any `pyspark.sql.types.DataType` class (or classes) you pass in. A category base class (e.g. `T.NumericType`) matches every concrete subclass of it too -- `by_dtype([T.NumericType])` matches integer, floating-point, *and* decimal columns in one call.
+
+``` python
+df.select(scs.by_dtype([T.StringType]))
+```
+
+| name  | region |
+|-------|--------|
+| north | east   |
+| south | west   |
+
+### `string()`
+
+``` python
+df.select(scs.string())
+```
+
+| name  | region |
+|-------|--------|
+| north | east   |
+| south | west   |
+
+### `boolean()`
+
+``` python
+df.select(scs.boolean())
+```
+
+| isactive |
+|----------|
+| true     |
+| false    |
+
+### `binary()`
+
+There are no binary columns in this DataFrame, so this raises an error by default rather than returning nothing -- see [Handling zero matches](#handling-zero-matches) below.
+
+### `integer()`
+
+Fixed-width integer columns (tinyint/smallint/int/bigint).
+
+``` python
+df.select(scs.integer())
+```
+
+| id  | totalscore | score2024 |
+|-----|------------|-----------|
+| 1   | 88         | 910       |
+| 2   | 95         | 875       |
+
+### `floats()`
+
+Floating-point columns (float/double).
+
+``` python
+df.select(scs.floats())
+```
+
+| elevation |
+|-----------|
+| 120.5     |
+| 340.0     |
+
+### `numeric()`
+
+Every fixed-width numeric column (integer or floating-point).
+
+``` python
+df.select(scs.numeric())
+```
+
+| id  | elevation | totalscore | score2024 |
+|-----|-----------|------------|-----------|
+| 1   | 120.5     | 88         | 910       |
+| 2   | 340.0     | 95         | 875       |
+
+### `date()`
+
+``` python
+df.select(scs.date())
+```
+
+| signupdate |
+|------------|
+| 2024-01-15 |
+| 2024-03-22 |
+
+### `datetime_()`
+
+``` python
+df.select(scs.datetime_())
+```
+
+| lastseen            |
+|---------------------|
+| 2024-06-01 08:30:00 |
+| 2024-06-02 14:45:00 |
+
+### `temporal()`
+
+Date or timestamp columns.
+
+``` python
+df.select(scs.temporal())
+```
+
+| signupdate | lastseen            |
+|------------|---------------------|
+| 2024-01-15 | 2024-06-01 08:30:00 |
+| 2024-03-22 | 2024-06-02 14:45:00 |
+
+------------------------------------------------------------------------
+
+## Position-based selectors
+
+### `by_index(*indexes)`
+
+Selects one or more columns by position. Negative positions work the same way Python indexing does (`-1` is the last column), and a `range(...)` can be mixed in with plain integers to grab a span in one call.
+
+``` python
+df.select(scs.by_index(range(0, 1), 3))
+```
+
+| id  | name  | elevation |
+|-----|-------|-----------|
+| 1   | north | 120.5     |
+| 2   | south | 340.0     |
+
+### `first()`
+
+``` python
+df.select(scs.first())
+```
+
+| id  |
+|-----|
+| 1   |
+| 2   |
+
+### `last()`
+
+``` python
+df.select(scs.last())
+```
+
+| lastseen            |
+|---------------------|
+| 2024-06-01 08:30:00 |
+| 2024-06-02 14:45:00 |
+
+### Handle column not in range error {#handle-column-not-in-range-error}
+
+By default, `by_index` raises if any requested position is out of range for the DataFrame (this DataFrame has 9 columns, at positions 0-8):
+
+``` python
+df.select(scs.by_index(9))
+# raises pyspark.errors.PySparkValueError -- position 9 is out of range
+```
+
+Passing `strict_index_bounds=False` skips any out-of-range position instead of raising, so only the in-range positions are matched:
+
+``` python
+df.select(scs.by_index(0, 9, strict_index_bounds=False))
+```
+
+| id  |
+|-----|
+| 1   |
+| 2   |
+
+> Note: if every requested position ends up out of range, `strict_index_bounds=False` alone still isn't enough to avoid an error -- the selector would then match zero columns, which raises on its own (see [Handling zero matches](#handling-zero-matches)). To allow that case too, also pass `require_col_match=False`.
+
+------------------------------------------------------------------------
+
+## Name-based selectors
+
+All name-based matching below is case-insensitive -- `scs.by_name("REGION")` matches a column named `region` just as well as `scs.by_name("region")` does, and column names may contain a literal dot (e.g. `meta.source`) without being mistaken for a nested field.
+
+### `matches(pattern)`
+
+Selects columns whose name matches a regular expression.
+
+``` python
+df.select(scs.matches("score"))
+```
+
+| totalscore | score2024 |
+|------------|-----------|
+| 88         | 910       |
+| 95         | 875       |
+
+`matches` takes any pattern the `re` module can compile, not just a plain substring -- anchors, character classes, and alternation all work:
+
+``` python
+df.select(scs.matches(r"^s.*[aeiou]$|\d"))
+```
+
+| signupdate | score2024 |
+|------------|-----------|
+| 2024-01-15 | 910       |
+| 2024-03-22 | 875       |
+
+This pattern matches a column name if it either starts with `s` and ends in a vowel (`signupdate`), or contains a digit anywhere in the name (`score2024`).
+
+### `starts_with(*prefixes)`
+
+``` python
+df.select(scs.starts_with("e"))
+```
+
+| elevation |
+|-----------|
+| 120.5     |
+| 340.0     |
+
+### `ends_with(*suffixes)`
+
+``` python
+df.select(scs.ends_with("date"))
+```
+
+| signupdate |
+|------------|
+| 2024-01-15 |
+| 2024-03-22 |
+
+### `contains(*substrings)`
+
+``` python
+df.select(scs.contains("2024"))
+```
+
+| score2024 |
+|-----------|
+| 910       |
+| 875       |
+
+### `by_name(*names)`
+
+Selects an exact list of column names.
+
+``` python
+df.select(scs.by_name("id", "region"))
+```
+
+| id  | region |
+|-----|--------|
+| 1   | east   |
+| 2   | west   |
+
+### `exclude(*names)`
+
+Selects every column except the given names.
+
+``` python
+df.select(scs.exclude("region"))
+```
+
+| id | name | elevation | totalscore | isactive | signupdate | score2024 | lastseen |
+|---------|---------|---------|---------|---------|---------|---------|---------|
+| 1 | north | 120.5 | 88 | true | 2024-01-15 | 910 | 2024-06-01 08:30:00 |
+| 2 | south | 340.0 | 95 | false | 2024-03-22 | 875 | 2024-06-02 14:45:00 |
+
+### `alpha()`
+
+Column names made up only of letters. `score2024` is the only column name containing a digit, so it's the only one excluded here.
+
+``` python
+df.select(scs.alpha())
+```
+
+| id  | name  | region | elevation | totalscore | isactive | signupdate | lastseen            |
+|---------|---------|---------|---------|---------|---------|---------|---------|
+| 1   | north | east   | 120.5     | 88         | true     | 2024-01-15 | 2024-06-01 08:30:00 |
+| 2   | south | west   | 340.0     | 95         | false    | 2024-03-22 | 2024-06-02 14:45:00 |
+
+### `alphanumeric()`
+
+Column names made up only of letters and/or digits -- every column name in this DataFrame qualifies, including `score2024`.
+
+``` python
+df.select(scs.alphanumeric())
+```
+
+Returns every column, identical to the example data table above.
+
+### `all()`
+
+``` python
+df.select(scs.all())
+```
+
+Returns every column, identical to the example data table above.
+
+------------------------------------------------------------------------
+
+## Combining selectors
+
+Selectors support set-style operators so more specific selections can be built without listing columns by hand. Combined selectors always resolve columns in the same order they appear in the underlying DataFrame.
+
+### Union (`|`)
+
+``` python
+df.select(scs.string() | scs.starts_with("e"))
+```
+
+| name  | region | elevation |
+|-------|--------|-----------|
+| north | east   | 120.5     |
+| south | west   | 340.0     |
+
+### Intersection (`&`)
+
+``` python
+df.select(scs.numeric() & scs.matches("score"))
+```
+
+| totalscore | score2024 |
+|------------|-----------|
+| 88         | 910       |
+| 95         | 875       |
+
+### Difference (`-`)
+
+``` python
+df.select(scs.all() - scs.numeric())
+```
+
+| name  | region | isactive | signupdate | lastseen            |
+|-------|--------|----------|------------|---------------------|
+| north | east   | true     | 2024-01-15 | 2024-06-01 08:30:00 |
+| south | west   | false    | 2024-03-22 | 2024-06-02 14:45:00 |
+
+### Symmetric difference (`^`)
+
+Columns matched by exactly one side, not both.
+
+``` python
+df.select(scs.contains("score") ^ scs.numeric())
+```
+
+| id  | elevation |
+|-----|-----------|
+| 1   | 120.5     |
+| 2   | 340.0     |
+
+### Complement (`~`)
+
+``` python
+df.select(~scs.temporal())
+```
+
+| id  | name  | region | elevation | totalscore | isactive | score2024 |
+|-----|-------|--------|-----------|------------|----------|-----------|
+| 1   | north | east   | 120.5     | 88         | true     | 910       |
+| 2   | south | west   | 340.0     | 95         | false    | 875       |
+
+------------------------------------------------------------------------
+
+## Chaining transforms onto a selector
+
+A transform can be chained onto a selector before it's resolved. The transform is applied to every column the selector matches, and the original column name is preserved in the result.
+
+### `.upper()`
+
+``` python
+df.select(scs.string().upper())
+```
+
+| name  | region |
+|-------|--------|
+| NORTH | EAST   |
+| SOUTH | WEST   |
+
+### `.cast(...)`
+
+``` python
+df.select(scs.floats().cast("string"))
+```
+
+| elevation |
+|-----------|
+| 120.5     |
+| 340.0     |
+
+The values look the same, but `elevation` is now a `string` column instead of a `double` column.
+
+### Arithmetic
+
+``` python
+df.select(scs.integer() + 1)
+```
+
+| id  | totalscore | score2024 |
+|-----|------------|-----------|
+| 2   | 89         | 911       |
+| 3   | 96         | 876       |
+
+Because every column function available in PySpark is also available as a chained method on a selector, this works with things like `.substr(...)`, `.isNull()`, `.between(...)`, and comparison operators too, not just a fixed list of built-ins.
+
+### Renaming columns:
+
+`.prefix(...)`, `.suffix(...)`, and `.map_alias(...)` Three functions used to modify a columns name.
+
+#### `.prefix(prefix)`
+
+Prepends `prefix` to every matched column's name.
+
+``` python
+df.select(scs.by_name("id", "region").prefix("src_"))
+```
+
+| src_id | src_region |
+|--------|------------|
+| 1      | east       |
+| 2      | west       |
+
+#### `.suffix(suffix)`
+
+Appends `suffix` to every matched column's name.
+
+``` python
+df.select(scs.numeric().suffix("_raw"))
+```
+
+| id_raw | elevation_raw | totalscore_raw | score2024_raw |
+|--------|---------------|----------------|---------------|
+| 1      | 120.5         | 88             | 910           |
+| 2      | 340.0         | 95             | 875           |
+
+#### `.map_alias(func)`
+
+Renames every matched column by passing its current name to `func`, a one-argument function returning the new name. `.prefix(...)` and `.suffix(...)` are both just `.map_alias(...)` with a prepend/append built in -- `.map_alias(...)` is for anything more custom.
+
+``` python
+df.select(scs.by_name("name", "region").map_alias(lambda n: n.upper()))
+```
+
+| NAME  | REGION |
+|-------|--------|
+| north | east   |
+| south | west   |
+
+Note this renames the *columns*, not the *values* -- compare to `.upper()` above, which uppercases the values but keeps the column names unchanged.
+
+------------------------------------------------------------------------
+
+## Using selectors with row filtering
+
+A selector used inside a comparison becomes a row condition instead of a column pick, and conditions built this way can be combined with `&`, `|`, and `~` the same way plain columns can. `filter`/`where` still return every original column for the matching rows -- only the relevant column(s) are shown below for readability.
+
+### A single condition
+
+``` python
+df.filter(scs.by_name("totalscore") > 90)
+```
+
+| id  | totalscore |
+|-----|------------|
+| 2   | 95         |
+
+*(the row for `id=1` is dropped, since `totalscore` there is 88)*
+
+### Combining conditions with `&`
+
+``` python
+df.filter((scs.by_name("totalscore") > 90) & (scs.by_name("elevation") > 200))
+```
+
+| id  | totalscore | elevation |
+|-----|------------|-----------|
+| 2   | 95         | 340.0     |
+
+------------------------------------------------------------------------
+
+## Using selectors with other DataFrame operations {#using-selectors-with-other-dataframe-operations}
+
+### `select` with multiple selectors
+
+``` python
+df.select(scs.string(), scs.integer())
+```
+
+| name  | region | id  | totalscore | score2024 |
+|-------|--------|-----|------------|-----------|
+| north | east   | 1   | 88         | 910       |
+| south | west   | 2   | 95         | 875       |
+
+### `groupBy` / `agg`
+
+``` python
+df.groupBy(scs.by_name("region")).agg(scs.integer().sum())
+```
+
+| region | sum(id) | sum(totalscore) | sum(score2024) |
+|--------|---------|-----------------|----------------|
+| east   | 1       | 88              | 910            |
+| west   | 2       | 95              | 875            |
+
+### `drop`
+
+``` python
+df.drop(scs.by_name("lastseen", "signupdate"))
+```
+
+Removes `lastseen` and `signupdate`, leaving `id`, `name`, `region`, `elevation`, `totalscore`, `isactive`, `score2024`.
+
+### `sort` / `orderBy`
+
+``` python
+df.sort(scs.by_name("elevation").desc())
+```
+
+| id  | elevation |
+|-----|-----------|
+| 2   | 340.0     |
+| 1   | 120.5     |
+
+*(every other original column is still present in the result; only `elevation` is shown here for readability)*
+
+### `withColumns` with multiple selector-driven mutations
+
+``` python
+df.withColumns(
+    scs.floats().cast("string"),
+    scs.string().upper(),
+)
+```
+
+| id  | name  | region | elevation |
+|-----|-------|--------|-----------|
+| 1   | NORTH | EAST   | 120.5     |
+| 2   | SOUTH | WEST   | 340.0     |
+
+Both mutations happen in the same call: every double column is cast to string, and every string column is uppercased.
+
+------------------------------------------------------------------------
+
+## Checking whether something is a selector
+
+``` python
+scs.is_selector(scs.numeric())  # True
+scs.is_selector("totalscore")  # False
+```
+
+------------------------------------------------------------------------
+
+## Handling zero matches {#handling-zero-matches}
+
+By default, if a selector resolves to zero columns against a given DataFrame, it raises an error rather than silently returning nothing. This also applies to combined selectors -- for example, subtracting a selector that covers every matching column from itself will raise, since the result has nothing left in it.
+
+``` python
+df.select(scs.binary())
+# raises pyspark.errors.PySparkValueError -- zero columns matched
+```
+
+Every selector function accepts a `require_col_match` keyword argument, which defaults to `True`. Passing `require_col_match=False` opts out of this check for that particular selector, allowing a zero-column match to pass through silently instead of raising:
+
+``` python
+df.select(scs.binary(require_col_match=False))
+# returns a result with zero columns instead of raising
+```
+
+This also applies to `by_dtype`, `by_index`, and `matches` directly:
+
+``` python
+df.select(scs.all() - scs.by_index(range(0, 8)))
+# There are 9 total columns in the df.
+# As such the above column selector expression returns all columns - all columns.
+# as such no columns of data are returned.
+# Result: raises pyspark.errors.PySparkValueError -- zero columns matched
+```
+
+To bypass the no columns returned error, see the below examples. All column selector functions have access to require_col_match by_index is the only one that offers access to strict_index_bounds as shown in [Handle column not in range error](#handle-column-not-in-range-error)
+
+``` python
+df.select(scs.by_dtype([T.BinaryType], require_col_match=False))
+df.select(scs.by_index(9, strict_index_bounds=False, require_col_match=False))
+df.select(scs.matches("no_such_pattern", require_col_match=False))
+```
+
+## Author
+
+PySparkSelectors was created in 2026 by Trevor A. Michel.
+
+Built with [Cookiecutter](https://github.com/cookiecutter/cookiecutter) and the [audreyfeldroy/cookiecutter-pypackage](https://github.com/audreyfeldroy/cookiecutter-pypackage) project template.

pyspark_column_selectors-0.1.3.dist-info/RECORD

@@ -0,0 +1,13 @@
+PySparkSelectors/__init__.py,sha256=E22iHgXsvwBoswHdxVhsBqPHjmiHfgOHn9aGHZjZofk,7196
+PySparkSelectors/__main__.py,sha256=Qd-f8z2Q2vpiEP2x6PBFsJrpACWDVxFKQk820MhFmHo,59
+PySparkSelectors/cli.py,sha256=HbigdTpEa4nVpBOmv3Yo5TvzEufBb0RU8jNCxike3OA,424
+PySparkSelectors/models.py,sha256=wlotfpDGXK2yuMiBn8_JhGkhYoYyHNEYS_T5f_psBs0,60688
+PySparkSelectors/py.typed,sha256=8PjyZ1aVoQpRVvt71muvuq5qE-jTFZkK-GLHkhdebmc,26
+PySparkSelectors/spark_overrides.py,sha256=SzlKR4ZneHzI-41MqxdiGe4y8w4tdBhy4KKLmXBD3lM,27100
+PySparkSelectors/user_functions.py,sha256=d5EjOmXUQ6bkpgJNCWnU3lvjw9OALyt3b_0k5k3ppPA,24535
+PySparkSelectors/utils.py,sha256=1SAf2SUXuLxIYMxF7qnm2AHt3ALK-SDvsUNn8sVh0n8,11133
+pyspark_column_selectors-0.1.3.dist-info/METADATA,sha256=15WBT99qqy08od1DMuSBFR-nwfDv0swvMb6Rf4tYV-U,19572
+pyspark_column_selectors-0.1.3.dist-info/WHEEL,sha256=zOwg4jB6zX2kU910N-cMawjivD6tO8NEWvE12je1bVk,87
+pyspark_column_selectors-0.1.3.dist-info/entry_points.txt,sha256=lDrzJJqOS4wQeRcg4ImS80KEIgogVbR_xtLMHpL0n7U,62
+pyspark_column_selectors-0.1.3.dist-info/licenses/LICENSE,sha256=Oy-B_iHRgcSZxZolbI4ZaEVdZonSaaqFNzv7avQdo78,13936
+pyspark_column_selectors-0.1.3.dist-info/RECORD,,

pyspark_column_selectors-0.1.3.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.32.0
+Root-Is-Purelib: true
+Tag: py3-none-any

pyspark_column_selectors-0.1.3.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+PySparkSelectors = PySparkSelectors.cli:app