pyduck-janitor 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (25) hide show
  1. pyduck_janitor-0.2.0/LICENSE +21 -0
  2. pyduck_janitor-0.2.0/PKG-INFO +596 -0
  3. pyduck_janitor-0.2.0/README.md +550 -0
  4. pyduck_janitor-0.2.0/pyduck_janitor/__init__.py +167 -0
  5. pyduck_janitor-0.2.0/pyduck_janitor/cleaning_ops.py +1029 -0
  6. pyduck_janitor-0.2.0/pyduck_janitor/cleaning_ops_extended.py +1016 -0
  7. pyduck_janitor-0.2.0/pyduck_janitor/cleaning_ops_final.py +877 -0
  8. pyduck_janitor-0.2.0/pyduck_janitor/duck_janitor.py +3349 -0
  9. pyduck_janitor-0.2.0/pyduck_janitor/embeddings.py +694 -0
  10. pyduck_janitor-0.2.0/pyduck_janitor/extensions.py +191 -0
  11. pyduck_janitor-0.2.0/pyduck_janitor/text_ops.py +455 -0
  12. pyduck_janitor-0.2.0/pyduck_janitor.egg-info/PKG-INFO +596 -0
  13. pyduck_janitor-0.2.0/pyduck_janitor.egg-info/SOURCES.txt +23 -0
  14. pyduck_janitor-0.2.0/pyduck_janitor.egg-info/dependency_links.txt +1 -0
  15. pyduck_janitor-0.2.0/pyduck_janitor.egg-info/requires.txt +28 -0
  16. pyduck_janitor-0.2.0/pyduck_janitor.egg-info/top_level.txt +1 -0
  17. pyduck_janitor-0.2.0/pyproject.toml +83 -0
  18. pyduck_janitor-0.2.0/setup.cfg +4 -0
  19. pyduck_janitor-0.2.0/tests/test_cleaning_ops.py +155 -0
  20. pyduck_janitor-0.2.0/tests/test_duck_janitor.py +957 -0
  21. pyduck_janitor-0.2.0/tests/test_embeddings.py +234 -0
  22. pyduck_janitor-0.2.0/tests/test_extensions.py +17 -0
  23. pyduck_janitor-0.2.0/tests/test_pyjanitor_aliases.py +754 -0
  24. pyduck_janitor-0.2.0/tests/test_text_ops.py +327 -0
  25. pyduck_janitor-0.2.0/tests/test_validation_and_edges.py +481 -0
pyduck_janitor-0.2.0/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 EzraAir555
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
pyduck_janitor-0.2.0/PKG-INFO ADDED
@@ -0,0 +1,596 @@
1
+ Metadata-Version: 2.4
2
+ Name: pyduck-janitor
3
+ Version: 0.2.0
4
+ Summary: DuckDB-backed pyjanitor for high-performance data cleaning, text search, and vector similarity on large datasets
5
+ Author-email: EzraAir555 <ezraair555@gmail.com>
6
+ License-Expression: MIT
7
+ Project-URL: Homepage, https://github.com/ezraair555/pyduck-janitor
8
+ Project-URL: Documentation, https://github.com/ezraair555/pyduck-janitor/tree/main/docs
9
+ Project-URL: Repository, https://github.com/ezraair555/pyduck-janitor
10
+ Keywords: duckdb,janitor,data cleaning,pandas,data preprocessing,etl
11
+ Classifier: Development Status :: 3 - Alpha
12
+ Classifier: Intended Audience :: Science/Research
13
+ Classifier: Programming Language :: Python :: 3
14
+ Classifier: Programming Language :: Python :: 3.9
15
+ Classifier: Programming Language :: Python :: 3.10
16
+ Classifier: Programming Language :: Python :: 3.11
17
+ Classifier: Programming Language :: Python :: 3.12
18
+ Classifier: Topic :: Scientific/Engineering :: Mathematics
19
+ Requires-Python: >=3.9
20
+ Description-Content-Type: text/markdown
21
+ License-File: LICENSE
22
+ Requires-Dist: duckdb>=0.9.0
23
+ Requires-Dist: pandas>=1.5.0
24
+ Requires-Dist: numpy>=1.20.0
25
+ Provides-Extra: dev
26
+ Requires-Dist: pytest>=7.0.0; extra == "dev"
27
+ Requires-Dist: pytest-cov>=3.0.0; extra == "dev"
28
+ Requires-Dist: black>=22.0.0; extra == "dev"
29
+ Requires-Dist: flake8>=4.0.0; extra == "dev"
30
+ Requires-Dist: mypy>=0.950; extra == "dev"
31
+ Requires-Dist: ruff>=0.6.0; extra == "dev"
32
+ Requires-Dist: openpyxl>=3.1; extra == "dev"
33
+ Requires-Dist: pyarrow>=15.0; extra == "dev"
34
+ Provides-Extra: io
35
+ Requires-Dist: openpyxl>=3.1; extra == "io"
36
+ Requires-Dist: pyarrow>=15.0; extra == "io"
37
+ Provides-Extra: vss
38
+ Requires-Dist: sentence-transformers>=2.2; extra == "vss"
39
+ Requires-Dist: huggingface_hub>=0.20; extra == "vss"
40
+ Provides-Extra: embeddings
41
+ Requires-Dist: pyduck-janitor-embeddings>=0.1; extra == "embeddings"
42
+ Requires-Dist: sentence-transformers>=2.2; extra == "embeddings"
43
+ Requires-Dist: huggingface_hub>=0.20; extra == "embeddings"
44
+ Provides-Extra: text
45
+ Dynamic: license-file
46
+
47
+ # pyduck-janitor
48
+
49
+ <p align="center">
50
+ <img src="docs/duck_janitor_logo.jpg" alt="pyduck-janitor logo" width="200" style="border-radius: 10%;"/>
51
+ </p>
52
+
53
+ **DuckDB-backed pyjanitor for high-performance data cleaning on large datasets**
54
+
55
+ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
56
+ [![Status: pre-release](https://img.shields.io/badge/status-pre--release-orange.svg)](https://github.com/ezraair555/pyduck-janitor)
57
+
58
+ The objective of this package is to perform data cleaning using an expressive grammar that coheres with the tidyverse design framework, but powered by [DuckDB](https://duckdb.org/) for high-performance execution on large datasets. The package is centered around data cleaning verbs, supplemented with many utilities for data transformation and manipulation.
59
+
60
+ ## Overview
61
+
62
+ pyduck-janitor provides a method-chaining API for data cleaning operations that mirrors [pyjanitor](https://pyjanitor-devs.github.io/pyjanitor/), but uses DuckDB as the backend for:
63
+
64
+ - **Speed**: DuckDB's vectorized execution engine accelerates data cleaning operations
65
+ - **Scalability**: Process datasets larger than memory by working directly with Parquet, CSV, or other file formats
66
+ - **Lazy evaluation**: Build complex cleaning pipelines that execute efficiently
67
+ - **Drop-in replacement**: Use familiar pyjanitor syntax with automatic DuckDB optimization
68
+
69
+ ### The Main Verbs
70
+
71
+ The core functionality of pyduck-janitor is organized around several main groups of verbs:
72
+
73
+ 1. **`clean_names()`** - Standardize column names to a consistent format
74
+ 2. **`filter_on()`, `filter_string()`** - Filter rows based on conditions or string patterns
75
+ 3. **`select_columns()`, `select_rows()`** - Select specific columns or rows
76
+ 4. **`add_column()`, `remove_columns()`, `rename_column()`** - Modify columns
77
+ 5. **`dropna()`, `remove_empty()`** - Handle missing data
78
+ 6. **`coalesce()`, `fill()`, `fill_empty()`** - Impute missing values
79
+ 7. **`encode_categorical()`, `get_dummies()`** - Encode categorical variables
80
+ 8. **`transform_column()`, `transform_columns()`** - Transform column values
81
+ 9. **`case_when()`, `find_replace()`** - Conditional transformations
82
+ 10. **`pivot_wider()`, `pivot_longer()`** - Reshape data
83
+ 11. **`groupby_agg()`, `groupby_topk()`** - Grouped operations
84
+
85
+ ## Installation
86
+
87
+ Install the released package from PyPI:
88
+
89
+ ```bash
90
+ pip install pyduck-janitor
91
+ ```
92
+
93
+ For Excel and Parquet I/O helpers, install the optional engines with
94
+ `pip install pyduck-janitor[io]`.
95
+
96
+ For development from a checkout, use the editable install. The `[dev]` extra
97
+ pulls in the test and lint dependencies used by `pytest`, `ruff`, and the
98
+ API-doc generator:
99
+
100
+ ```bash
101
+ git clone https://github.com/ezraair555/pyduck-janitor.git
102
+ cd pyduck-janitor
103
+ pip install -e ".[dev]"
104
+ ```
105
+
106
+ ## Quick Start
107
+
108
+ ```python
109
+ import pandas as pd
110
+ from pyduck_janitor import DuckJanitor
111
+
112
+ # Load your data
113
+ df = DuckJanitor.from_pandas(pd.DataFrame({
114
+ 'SalesMonth': ['Jan', 'Feb', 'Mar', 'April'],
115
+ 'Company1': [150.0, 200.0, 300.0, 400.0],
116
+ 'Company2': [180.0, 250.0, None, 500.0],
117
+ 'Company3': [400.0, 500.0, 600.0, 675.0]
118
+ }))
119
+
120
+ # Build a cleaning pipeline
121
+ result = (
122
+ df
123
+ .clean_names()
124
+ .remove_columns(['company1'])
125
+ .dropna(subset=['company2', 'company3'])
126
+ .rename_column('company2', 'amazon')
127
+ .add_column('google', [450.0, 550.0, 800.0])
128
+ .collect()
129
+ )
130
+
131
+ print(result)
132
+ ```
133
+
134
+ ### Select DSL (pyjanitor-compatible)
135
+
136
+ ```python
137
+ from pyduck_janitor import DuckJanitor, DropLabel
138
+
139
+ dj = DuckJanitor.from_pandas(pd.DataFrame({
140
+ 'sales_month': ['Jan', 'Feb'],
141
+ 'company2': [180.0, 250.0],
142
+ 'company3': [400.0, 500.0],
143
+ 'notes': ['x', 'y'],
144
+ }))
145
+
146
+ dj.select_columns('sales_month, company2') # comma-string
147
+ dj.select_columns('company*') # glob expansion
148
+ dj.select_columns('re:^company') # regex
149
+ dj.select_columns(['company*', DropLabel('company3')]) # exclude one column
150
+ ```
151
+
152
+ ## Supported Functions
153
+
154
+ pyduck-janitor implements the **complete pyjanitor documented API** (94/94 functions — verified against the pyjanitor API reference) plus 18 DuckDB-specific extensions, exposed as **109 chainable methods** on `DuckJanitor`.
155
+
156
+ > 📖 **Full function-by-function reference with verified examples:** [`docs/api/functions.md`](docs/api/functions.md) — description, signature, parameters, returns, raises, and a runnable example for every function (modeled on [pyjanitor's API functions page](https://pyjanitor-devs.github.io/pyjanitor/api/functions/)).
157
+
158
+ ### Core cleaning verbs (`cleaning_ops.py`)
159
+ - [`clean_names()`](docs/api/functions.md#clean_names) - Clean column names
160
+ - [`remove_columns()`](docs/api/functions.md#remove_columns) - Remove columns
161
+ - [`add_column()`](docs/api/functions.md#add_column) / [`add_columns()`](docs/api/functions.md#add_columns) - Add a new column (single or dict form)
162
+ - [`rename_column()`](docs/api/functions.md#rename_column) / [`rename_columns()`](docs/api/functions.md#rename_columns) - Rename a column
163
+ - [`dropna()`](docs/api/functions.md#dropna) - Drop rows with NA values
164
+ - [`remove_empty()`](docs/api/functions.md#remove_empty) - Remove empty rows/columns
165
+ - [`filter_column()`](docs/api/functions.md#filter_column) / [`filter_column_isin()`](docs/api/functions.md#filter_column_isin) - Filter by column condition / IS IN list
166
+ - [`filter_on()`](docs/api/functions.md#filter_on) - Filter with SQL-like criteria
167
+ - [`filter_string()`](docs/api/functions.md#filter_string) - Filter by substring
168
+ - [`coalesce()`](docs/api/functions.md#coalesce) - Merge columns
169
+ - [`encode_categorical()`](docs/api/functions.md#encode_categorical) - Encode as categorical
170
+ - [`get_dummies()`](docs/api/functions.md#get_dummies) - One-hot encode
171
+ - [`select_columns()`](docs/api/functions.md#select_columns) / [`select()`](docs/api/functions.md#select) - Select columns (supports comma-strings, globs, `re:` regex, and [`DropLabel`](docs/api/functions.md#DropLabel))
172
+ - [`select_rows()`](docs/api/functions.md#select_rows) - Select rows
173
+ - [`transform_column()`](docs/api/functions.md#transform_column) / [`transform_columns()`](docs/api/functions.md#transform_columns) - Transform one or many columns
174
+
175
+ ### Extended verbs (`cleaning_ops_extended.py`)
176
+ - [`bin_numeric()`](docs/api/functions.md#bin_numeric) - Bin numeric column
177
+ - [`change_type()`](docs/api/functions.md#change_type) - Change column type
178
+ - [`concatenate_columns()`](docs/api/functions.md#concatenate_columns) - Join columns
179
+ - [`deconcatenate_column()`](docs/api/functions.md#deconcatenate_column) - Split column
180
+ - [`drop_constant_columns()`](docs/api/functions.md#drop_constant_columns) - Remove constant columns
181
+ - [`fill()`](docs/api/functions.md#fill) / [`fill_direction()`](docs/api/functions.md#fill_direction) - Fill missing values (forward/backward)
182
+ - [`fill_empty()`](docs/api/functions.md#fill_empty) - Fill empty strings
183
+ - [`flag_nulls()`](docs/api/functions.md#flag_nulls) - Flag null values
184
+ - [`limit_column_characters()`](docs/api/functions.md#limit_column_characters) - Truncate column names
185
+ - [`min_max_scale()`](docs/api/functions.md#min_max_scale) - Scale to [0,1]
186
+ - [`groupby_agg()`](docs/api/functions.md#groupby_agg) - Group and aggregate
187
+ - [`groupby_topk()`](docs/api/functions.md#groupby_topk) - Top k per group
188
+ - [`case_when()`](docs/api/functions.md#case_when) - Conditional logic
189
+ - [`currency_column_to_numeric()`](docs/api/functions.md#currency_column_to_numeric) - Parse currency
190
+ - [`convert_date()`](docs/api/functions.md#convert_date) / [`convert_to_date()`](docs/api/functions.md#convert_to_date) / [`convert_to_datetime()`](docs/api/functions.md#convert_to_datetime) - Convert to date/datetime
191
+ - [`convert_unix_date()`](docs/api/functions.md#convert_unix_date), [`convert_excel_date()`](docs/api/functions.md#convert_excel_date), [`convert_matlab_date()`](docs/api/functions.md#convert_matlab_date) - Numeric date conversions
192
+ - [`truncate_datetime()`](docs/api/functions.md#truncate_datetime) / [`truncate_datetime_dataframe()`](docs/api/functions.md#truncate_datetime_dataframe) - Truncate datetime
193
+ - [`pivot_wider()`](docs/api/functions.md#pivot_wider) / [`pivot_longer()`](docs/api/functions.md#pivot_longer) - Reshape wide/long
194
+
195
+ ### Hybrid verbs (`cleaning_ops_final.py`)
196
+ - [`conditional_join()`](docs/api/functions.md#conditional_join) - Join with condition
197
+ - [`get_dupes()`](docs/api/functions.md#get_dupes) - Find duplicate rows
198
+ - [`dropnotnull()`](docs/api/functions.md#dropnotnull) - Drop non-null values
199
+ - [`expand_column()`](docs/api/functions.md#expand_column) - Expand delimited column
200
+ - [`impute()`](docs/api/functions.md#impute) - Impute missing values
201
+ - [`jitter()`](docs/api/functions.md#jitter) - Add noise to values
202
+ - [`label_encode()`](docs/api/functions.md#label_encode) - Encode as integers
203
+ - [`find_replace()`](docs/api/functions.md#find_replace) - Replace values
204
+ - [`count_cumulative_unique()`](docs/api/functions.md#count_cumulative_unique) - Count unique values
205
+ - [`complete()`](docs/api/functions.md#complete) - Complete missing combinations
206
+ - [`also()`](docs/api/functions.md#also) - Apply multiple operations
207
+ - [`alias()`](docs/api/functions.md#alias) - Create column aliases
208
+ - [`mutate()`](docs/api/functions.md#mutate) / [`assign()`](docs/api/functions.md#assign) / [`ungroup()`](docs/api/functions.md#ungroup) - Add/modify columns, tidyverse-style verbs
209
+ - [`drop_duplicate_columns()`](docs/api/functions.md#drop_duplicate_columns) - Remove duplicate columns
210
+ - [`compare_df_cols()`](docs/api/functions.md#compare_df_cols) / [`compare_df_cols_same()`](docs/api/functions.md#compare_df_cols_same) - Compare column contents/shape
211
+ - [`join_apply()`](docs/api/functions.md#join_apply) - Apply function to joined data
212
+ - [`process_text()`](docs/api/functions.md#process_text) - Text processing
213
+
214
+ ### pyjanitor parity methods (v0.2.0)
215
+
216
+ These were added in v0.2.0 to reach 100% coverage of pyjanitor's documented API. Every function in this table has a pyjanitor counterpart — the second column shows how the pyduck-janitor method relates to it (same-name implementation or alias of another pyduck verb). Functions with **no** pyjanitor counterpart are listed separately under [DuckDB-only extensions](#duckdb-only-extensions) — none of them appear in this table.
217
+
218
+ | pyjanitor function | pyduck-janitor | Notes |
219
+ | --- | --- | --- |
220
+ | [`rename_columns`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.rename_columns.rename_columns) | alias of [`rename_column`](docs/api/functions.md#rename_column) | plural form |
221
+ | [`truncate_datetime_dataframe`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.truncate_datetime.truncate_datetime_dataframe) | alias of [`truncate_datetime`](docs/api/functions.md#truncate_datetime) |
222
+ | [`convert_to_date`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.convert_date.convert_to_date) / [`convert_to_datetime`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.convert_date.convert_to_datetime) | aliases of [`convert_date`](docs/api/functions.md#convert_date) | |
223
+ | [`convert_unix_date`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.convert_date.convert_unix_date) | same-name port of pyjanitor [`convert_unix_date`](docs/api/functions.md#convert_unix_date) | `TO_TIMESTAMP`, seconds/millis/micros |
224
+ | [`convert_excel_date`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.convert_date.convert_excel_date) | same-name port of pyjanitor [`convert_excel_date`](docs/api/functions.md#convert_excel_date) | Excel serial dates |
225
+ | [`convert_matlab_date`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.convert_date.convert_matlab_date) | same-name port of pyjanitor [`convert_matlab_date`](docs/api/functions.md#convert_matlab_date) | MATLAB datenums |
226
+ | [`excel_time_to_numeric`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.convert_date.excel_time_to_numeric) | same-name port of pyjanitor [`excel_time_to_numeric`](docs/api/functions.md#excel_time_to_numeric) | Excel time fraction → seconds |
227
+ | [`sas_numeric_to_date`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.convert_date.sas_numeric_to_date) | same-name port of pyjanitor [`sas_numeric_to_date`](docs/api/functions.md#sas_numeric_to_date) | SAS origin 1960-01-01 |
228
+ | [`to_datetime`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.to_datetime.to_datetime) | same-name port of pyjanitor [`to_datetime`](docs/api/functions.md#to_datetime) | DuckDB `strptime` cast |
229
+ | [`fill_direction`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.fill.fill_direction) | alias of [`fill`](docs/api/functions.md#fill) | |
230
+ | [`filter_column_isin`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.filter.filter_column_isin) | same-name port of pyjanitor [`filter_column_isin`](docs/api/functions.md#filter_column_isin) | quoted-column `IS IN` filter |
231
+ | [`filter_date`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.filter.filter_date) | same-name port of pyjanitor [`filter_date`](docs/api/functions.md#filter_date) | start/end date range filter |
232
+ | [`add_columns`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.add_columns.add_columns) | same-name port of pyjanitor [`add_columns`](docs/api/functions.md#add_columns) | dict of `{name: values}` |
233
+ | [`assign`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.mutate.assign) / [`ungroup`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.mutate.ungroup) | aliases of [`mutate`](docs/api/functions.md#mutate) / no-op | tidyverse naming |
234
+ | [`get_columns`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.select.get_columns) / [`get_index_labels`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.select.get_index_labels) | same-name ports of the pyjanitor select helpers | column introspection |
235
+ | [`move`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.move.move) / [`reorder_columns`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.reorder_columns.reorder_columns) | same-name ports of pyjanitor [`move`](docs/api/functions.md#move) / [`reorder_columns`](docs/api/functions.md#reorder_columns) | column placement verbs |
236
+ | [`row_to_names`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.row_to_names.row_to_names) | same-name port of pyjanitor [`row_to_names`](docs/api/functions.md#row_to_names) | promote a row to headers |
237
+ | [`rle_id`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.rle_id.rle_id) | same-name port of pyjanitor [`rle_id`](docs/api/functions.md#rle_id) | run-length ids via hash + window |
238
+ | [`factorize_columns`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.factorize_columns.factorize_columns) | same-name port of pyjanitor [`factorize_columns`](docs/api/functions.md#factorize_columns) | `DENSE_RANK` integer encoding |
239
+ | [`sort_naturally`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.sort_naturally.sort_naturally) | same-name port of pyjanitor [`sort_naturally`](docs/api/functions.md#sort_naturally) | human (non-lexicographic) sort |
240
+ | [`sort_column_value_order`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.sort_column_value_order.sort_column_value_order) | same-name port of pyjanitor [`sort_column_value_order`](docs/api/functions.md#sort_column_value_order) | explicit value ordering |
241
+ | [`update_where`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.update_where.update_where) | same-name port of pyjanitor [`update_where`](docs/api/functions.md#update_where) | conditional column update |
242
+ | [`unionize_dataframe_categories`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.utils.unionize_dataframe_categories) | same-name port of pyjanitor [`unionize_dataframe_categories`](docs/api/functions.md#unionize_dataframe_categories) | cross-relation VARCHAR alignment |
243
+ | [`shuffle`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.shuffle.shuffle) | same-name port of pyjanitor [`shuffle`](docs/api/functions.md#shuffle) | `ORDER BY random()` |
244
+ | [`toset`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.toset.toset) | same-name port of pyjanitor [`toset`](docs/api/functions.md#toset) | distinct values as a list |
245
+ | [`take_first`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.take_first.take_first) | same-name port of pyjanitor [`take_first`](docs/api/functions.md#take_first) | first N rows |
246
+ | [`round_to_fraction`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.round_to_fraction.round_to_fraction) | same-name port of pyjanitor [`round_to_fraction`](docs/api/functions.md#round_to_fraction) | snap to 1/denominator |
247
+ | [`scale_mad`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.scale_mad.scale_mad) | same-name port of pyjanitor [`scale_mad`](docs/api/functions.md#scale_mad) | median-abs-deviation scaling |
248
+ | [`cartesian_product`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.expand_grid.cartesian_product) | same-name port of pyjanitor [`cartesian_product`](docs/api/functions.md#cartesian_product) | cross join helper |
249
+ | [`then`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.then.then) | same-name port of pyjanitor [`then`](docs/api/functions.md#then) | chain callables |
250
+ | [`expand`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.expand_grid.expand) / [`expand_grid`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.expand_grid.expand_grid) | same-name ports of the pyjanitor expand family | distinct expansion / cross join grid |
251
+ | [`change_index_dtype`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.change_index_dtype.change_index_dtype) | same-name port of pyjanitor [`change_index_dtype`](docs/api/functions.md#change_index_dtype) | typed projection of a column |
252
+ | [`collapse_levels`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.collapse_levels.collapse_levels) | same-name port of pyjanitor [`collapse_levels`](docs/api/functions.md#collapse_levels) | concat-join helper |
253
+ | [`explode_index`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.explode_index.explode_index) | same-name port of pyjanitor [`explode_index`](docs/api/functions.md#explode_index) | regex-extract a parsed column |
254
+ | [`summarise`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.summarise.summarise) | same-name port of pyjanitor [`summarise`](docs/api/functions.md#summarise) | group-by aggregation helper |
255
+ | [`pivot_longer_spec`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.pivot.pivot_longer_spec) / [`pivot_wider_spec`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.pivot.pivot_wider_spec) | same-name ports of the pyjanitor `_spec` pivots | UNPIVOT / PIVOT spec forms |
256
+ | [`join_agg`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.conditional_join.join_agg) / [`get_join_indices`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.conditional_join.get_join_indices) | same-name ports of the pyjanitor conditional-join helpers | aggregated/non-equi join |
257
+ | [`select`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.select.select) | pyjanitor [`select`](docs/api/functions.md#select) folded into [`select_columns`](docs/api/functions.md#select_columns) + [`select()`](docs/api/functions.md#select) alias | comma-strings, globs, `re:` regex, [`DropLabel`](docs/api/functions.md#DropLabel) |
258
+ | [`DropLabel`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.select.DropLabel) | same-name port of pyjanitor [`DropLabel`](docs/api/functions.md#DropLabel) | select-DSL exclusion sentinel |
259
+ | [`patterns`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.utils.patterns) | same-name port of pyjanitor [`patterns`](docs/api/functions.md#patterns) | regex helper with `.compiled` |
260
+ | [`describe_class`](https://pyjanitor-devs.github.io/pyjanitor/api/functions/#janitor.functions.compare_df_cols.describe_class) | same-name port of pyjanitor [`describe_class`](docs/api/functions.md#describe_class) | column-type table (DESCRIBE-backed) |
261
+
262
+ ### DuckDB-only extensions (new — no pyjanitor equivalent)
263
+
264
+ These are the only truly *new* methods (no pyjanitor counterpart) — they exist because the backend is a live DuckDB connection rather than a pandas DataFrame:
265
+
266
+ - [`from_pandas()`](docs/api/functions.md#from_pandas), [`from_csv()`](docs/api/functions.md#from_csv), [`from_excel()`](docs/api/functions.md#from_excel), [`from_json()`](docs/api/functions.md#from_json), [`from_parquet()`](docs/api/functions.md#from_parquet), [`from_sql()`](docs/api/functions.md#from_sql) - Data source loaders
267
+ - [`sql()`](docs/api/functions.md#sql) - Escape hatch: raw SQL against the current relation (use `self` as the table name)
268
+ - [`explain()`](docs/api/functions.md#explain) - EXPLAIN plan for the current pipeline
269
+ - [`collect()`](docs/api/functions.md#collect) / [`head()`](docs/api/functions.md#head) - Materialize to pandas / preview rows
270
+ - [`get_shared_connection()`](docs/api/functions.md#get_shared_connection) - access to the underlying DuckDB connection
271
+
272
+ Plus the pandas-flavored bases that pyjanitor implements differently and pyduck implements natively (comparable intent, DuckDB-native implementation — documented in the module tables above): [`dropna`](docs/api/functions.md#dropna), [`fill`](docs/api/functions.md#fill), [`filter_column`](docs/api/functions.md#filter_column), [`convert_date`](docs/api/functions.md#convert_date), [`truncate_datetime`](docs/api/functions.md#truncate_datetime), [`get_dummies`](docs/api/functions.md#get_dummies).
273
+
274
+ ## Supported Data Sources
275
+
276
+ pyduck-janitor can work with data from:
277
+
278
+ - **In-memory pandas DataFrames** - Via `from_pandas()`
279
+ - **Parquet files** - Local or remote (S3, HTTP)
280
+ - **CSV files** - Local or remote
281
+ - **JSON files** - Local or remote
282
+ - **DuckDB databases** - Existing `.duckdb` files
283
+ - **SQL queries** - Custom SQL as input
284
+
285
+ ## Text & Similarity (v0.2.0)
286
+
287
+ Beyond structural cleaning, pyduck-janitor wraps three lazy-loaded DuckDB
288
+ extensions — `icu`, `fts`, and `vss` — for messy-text work at scale:
289
+
290
+ | Verb | Backend | What it does |
291
+ |---|---|---|
292
+ | `text_normalize()` | icu | Lowercase, accent strip, whitespace collapse |
293
+ | `search_text()` | fts | BM25-ranked full-text search |
294
+ | `keyword_filter()` | fts | Boolean contains (any/all phrases) |
295
+ | `build_fts_index()` / `drop_fts_index()` | fts | Index lifecycle |
296
+ | `embed_column()` | vss + sentence-transformers | Embed a text column |
297
+ | `build_vector_index()` / `vector_search()` | vss | HNSW kNN search |
298
+ | `fuzzy_dedupe()` | vss | Near-duplicate detection |
299
+ | `embed_install()` / `embed_list_installed()` / `embed_remove()` | — | Model cache management |
300
+
301
+ ```python
302
+ import pandas as pd
303
+ from pyduck_janitor import DuckJanitor, build_fts_index, search_text, text_normalize
304
+
305
+ df = pd.DataFrame({"text": ["The quick brown fox", "lazy dogs", "brown foxes"]})
306
+ dj = DuckJanitor.from_pandas(df)
307
+ dj = build_fts_index(dj, "text")
308
+
309
+ results = search_text(dj, "text", "fox quick", top_k=3)
310
+ # __pyduck_rowid text score
311
+ # 0 3 brown foxes 0.34
312
+ # 1 1 The quick brown fox 0.28
313
+ ```
314
+
315
+ Embedding models are never downloaded silently — `embed_install()` is the
316
+ explicit opt-in, and supports the bundled companion wheel (offline), any
317
+ HuggingFace model (`hf:org/model`), or a local path. Full guide:
318
+ [`docs/text_ops.md`](docs/text_ops.md).
319
+
320
+ ### Adding a HuggingFace model
321
+
322
+ Any sentence-transformers-compatible model from HuggingFace Hub works.
323
+ Pull it into the local cache by passing an `hf:`-prefixed identifier:
324
+
325
+ ```python
326
+ import pyduck_janitor as pj
327
+
328
+ # Public model — direct fetch
329
+ pj.embed_install("hf:sentence-transformers/all-MiniLM-L6-v2")
330
+ pj.embed_install("hf:BAAI/bge-small-en-v1.5") # better quality, 33M params
331
+ pj.embed_install("hf:BAAI/bge-base-en-v1.5") # bigger, ~110M params
332
+ pj.embed_install("hf:intfloat/multilingual-e5-small") # multilingual
333
+
334
+ # Pin a specific revision (sha or tag) for reproducibility
335
+ pj.embed_install("hf:org/model@sha256:abc123...")
336
+ pj.embed_install("hf:org/model@refs/pr/42")
337
+
338
+ # Gated / private models — set HF_TOKEN in your env, then call as usual
339
+ # export HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxx
340
+ pj.embed_install("hf:meta-llama/Llama-Embed-8B")
341
+
342
+ # Or a local directory you've prepared (no HuggingFace needed)
343
+ pj.embed_install("/opt/models/my-finetuned-encoder")
344
+ ```
345
+
346
+ Once installed, the model is available everywhere in your workflow:
347
+
348
+ ```python
349
+ dj = DuckJanitor.from_pandas(df)
350
+ dj = embed_column(dj, "text", model="hf:BAAI/bge-small-en-v1.5")
351
+ dj = build_vector_index(dj, metric="cosine")
352
+ hits = vector_search(dj, "lazy dog", model="hf:BAAI/bge-small-en-v1.5", top_k=5)
353
+ ```
354
+
355
+ Manage the cache over time:
356
+
357
+ ```python
358
+ pj.embed_list_installed()
359
+ # model size_human has_config has_weights
360
+ # 0 sentence-transformers/all-MiniLM-L6-v2 90.2 MB True True
361
+ # 1 BAAI/bge-small-en-v1.5 33.4 MB True True
362
+
363
+ pj.embed_remove("hf:BAAI/bge-small-en-v1.5")
364
+ ```
365
+
366
+ Override the cache location with `PYDUCK_EMBED_CACHE=/some/path` when you
367
+ need shared storage across venvs, CI runners, or read-only filesystems.
368
+
369
+ ## Key Features
370
+
371
+ ### Lazy Evaluation
372
+
373
+ Operations build a query plan without immediate execution. Use `.collect()` to execute:
374
+
375
+ ```python
376
+ result = df.clean_names().remove_empty().dropna().collect()
377
+ ```
378
+
379
+ ### Out-of-Core Processing
380
+
381
+ Work with datasets larger than RAM:
382
+
383
+ ```python
384
+ df = DuckJanitor.from_parquet('large_dataset.parquet')
385
+ result = df.clean_names().remove_empty().collect()
386
+ ```
387
+
388
+ ### Method Chaining
389
+
390
+ All methods return DuckJanitor objects, enabling fluent pipelines:
391
+
392
+ ```python
393
+ result = (
394
+ df
395
+ .clean_names()
396
+ .filter_on('age > 18')
397
+ .groupby_agg('gender', {'income': 'mean'})
398
+ .collect()
399
+ )
400
+ ```
401
+
402
+ ### SQL Interoperability
403
+
404
+ Mix janitor methods with custom SQL:
405
+
406
+ ```python
407
+ result = df.sql('SELECT * FROM self WHERE age > 18').collect()
408
+ ```
409
+
410
+ ### SQL Fragment Safety (0.1.3)
411
+
412
+ For methods that accept SQL fragments (for example `filter_on`, `filter_column`, `select_rows(criteria=...)`, `transform_column(func=...)`, `case_when`, and `change_type`), `pyduck-janitor` now rejects:
413
+
414
+ - Multi-statement fragments containing `;`
415
+ - SQL comments (`--`, `/* ... */`)
416
+ - Destructive DDL/DML keywords (`DROP`, `DELETE`, `UPDATE`, `INSERT`, etc.)
417
+
418
+ This keeps expression-based APIs usable while reducing accidental or unsafe query fragments.
419
+
420
+ ## API Comparison
421
+
422
+ ### Traditional pandas + pyjanitor
423
+
424
+ ```python
425
+ import pandas as pd
426
+ import janitor
427
+
428
+ df = pd.read_csv('large_file.csv')
429
+ df = (
430
+ df
431
+ .clean_names()
432
+ .remove_empty()
433
+ .dropna(subset=['col1', 'col2'])
434
+ )
435
+ ```
436
+
437
+ ### pyduck-janitor (faster, scalable)
438
+
439
+ ```python
440
+ from pyduck_janitor import DuckJanitor
441
+
442
+ df = DuckJanitor.from_csv('large_file.csv')
443
+ df = (
444
+ df
445
+ .clean_names()
446
+ .remove_empty()
447
+ .dropna(subset=['col1', 'col2'])
448
+ )
449
+ result = df.collect() # Explicit execution
450
+ ```
451
+
452
+ ## Examples
453
+
454
+ See the `examples/` directory for complete workflows:
455
+
456
+ - `examples/basic_cleaning.py` - Basic data cleaning pipeline
457
+ - `examples/large_dataset.py` - Out-of-core processing with Parquet
458
+ - `examples/sql_interop.py` - Mixing janitor methods with SQL
459
+ - `examples/comparison.py` - Performance comparison with pandas + pyjanitor
460
+
461
+ ## Architecture
462
+
463
+ pyduck-janitor works by:
464
+
465
+ 1. **Wrapping DuckDB relations** - Data is stored in DuckDB tables
466
+ 2. **Translating janitor methods** - Each method converts to DuckDB SQL
467
+ 3. **Lazy evaluation** - Operations build a query plan
468
+ 4. **Optimized execution** - DuckDB executes the entire pipeline efficiently
469
+ 5. **Pandas compatibility** - Results can be converted to pandas DataFrames
470
+
471
+ ### Hybrid Pattern
472
+
473
+ For operations that can't be pure SQL:
474
+
475
+ 1. **Materialize** - Convert DuckDB relation to pandas DataFrame
476
+ 2. **Apply** - Execute Python function
477
+ 3. **Re-wrap** - Create new DuckJanitor instance
478
+
479
+ ## Performance
480
+
481
+ pyduck-janitor provides significant speedups for:
482
+
483
+ - Large datasets (>1M rows)
484
+ - Complex cleaning pipelines
485
+ - Operations on disk-based data
486
+ - Column-wise transformations
487
+
488
+ Benchmark results vary by workload, but expect 2-10x speedups on typical data cleaning tasks.
489
+
490
+ ## Contributing
491
+
492
+ We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.
493
+
494
+
495
+ ## Changelog
496
+
497
+ ### 0.2.0 — 100% pyjanitor API parity
498
+
499
+ - **Full pyjanitor surface**: pyduck-janitor now covers **94/94 (100%)** of the
500
+ functions documented on the pyjanitor API reference, verified by a fresh
501
+ scan of pyjanitor's live docs.
502
+ - **~35 newly added chainable methods** on `DuckJanitor` — all ports of pyjanitor functions (same names except where noted as aliases), not pyduck-only inventions; the pyduck-only extensions are listed separately:
503
+ - Date conversions: `convert_unix_date`, `convert_excel_date`,
504
+ `convert_matlab_date`, `excel_time_to_numeric`, `sas_numeric_to_date`,
505
+ `to_datetime` (all with float-safe numeric parsing via `TO_TIMESTAMP`),
506
+ plus `convert_to_date` / `convert_to_datetime` aliases.
507
+ - Structural verbs: `move`, `reorder_columns`, `get_columns`,
508
+ `get_index_labels`, `row_to_names`, `collapse_levels`, `explode_index`,
509
+ `change_index_dtype`.
510
+ - Reshape/aggregation: `expand`, `expand_grid`, `summarise`,
511
+ `pivot_longer_spec` (UNPIVOT), `pivot_wider_spec` (PIVOT with a literal
512
+ value list), `join_agg`, `get_join_indices`.
513
+ - Data quality / encoding: `rle_id`, `factorize_columns`, `update_where`,
514
+ `unionize_dataframe_categories`, `scale_mad`, `round_to_fraction`.
515
+ - Row/column utilities: `shuffle`, `toset`, `take_first`,
516
+ `sort_naturally`, `sort_column_value_order`, `filter_date`,
517
+ `cartesian_product`, `then`.
518
+ - pyjanitor naming aliases: `rename_columns`, `truncate_datetime_dataframe`,
519
+ `fill_direction`, `filter_column_isin`, `add_columns`, `assign`, `ungroup`.
520
+ - **Select DSL**: `select_columns` now accepts comma-separated strings
521
+ (`"a, b, c"`), shell-globs (`"value*"`), and regex (`"re:^v_"`), matching
522
+ pyjanitor's `select` helper where it lives (under `select_columns`).
523
+ A thin `select()` alias is exposed; non-column kwargs raise
524
+ `NotImplementedError` (pyjanitor itself deprecates them).
525
+ - **pyjanitor helper surface**: `DropLabel` (select-DSL exclusion sentinel,
526
+ functional in mixed lists), `patterns` (regex helper), and
527
+ `describe_class()` (DESCRIBE-backed column-type table).
528
+ - **Test suite**: grew from 193 to **284 passing tests**
529
+ (+91 alias/DSL/parity tests in `tests/test_pyjanitor_aliases.py`).
530
+ - **Docs**: fixed stale example references and expanded the supported
531
+ functions section; parity analysis in
532
+ [`REVIEW_PYJANITOR_PARITY.md`](REVIEW_PYJANITOR_PARITY.md).
533
+
534
+ ### 0.1.3 — Validation hardening, audit documentation, and test expansion
535
+
536
+ - Added a full package audit report in `CODE_REVIEW.md`.
537
+ - Added stronger validation for SQL-fragment inputs and missing-column errors across cleaning modules.
538
+ - Added expanded edge-case and error-path tests in `tests/test_validation_and_edges.py`.
539
+ - Improved `join_apply` cross-connection handling and `DuckJanitor.sql()` identifier replacement behavior.
540
+ - Bumped package version to `0.1.3`.
541
+
542
+ ### 0.1.2 — Production-ready stabilization, pure SQL rewrites, and test expansion
543
+
544
+ - **Pure SQL Rewrites**: Rewrote `alias`, `complete`, and `drop_duplicate_columns` in 100% pure, out-of-core SQL to avoid in-memory materialization to Pandas.
545
+ - **API Expositions**: Exposed previously hidden hybrid and final functions (`drop_duplicate_columns`, `compare_df_cols`, `join_apply`, `process_text`, and `get_dupes`) directly as wrapper methods on `DuckJanitor`.
546
+ - **Bug Fixes**:
547
+ - Fixed syntax parser errors in `fill` by introducing physical row number index CTEs instead of nesting window functions.
548
+ - Added safe string literal quoting fallback to `add_column` and `filter_column` when passing raw string scalars.
549
+ - Resolved name-collision bugs in `clean_names` and `coalesce`.
550
+ - Implemented group-by partitioning support in `impute` using SQL window functions.
551
+ - Ensured operations like `fill_empty`, `currency_column_to_numeric`, and `convert_date` gracefully return NULLs instead of crashing.
552
+ - **Metadata Update**: Updated package version to `0.1.2` and author information.
553
+ - **Unit Test Suite**: Added 54 new test cases covering all edge cases, raising code coverage from **44% to 93%** (with 100% coverage on `duck_janitor.py`).
554
+ - **Logo Sticker**: Added a custom package logo sticker of a duck dressed as a janitor.
555
+
556
+ ### 0.1.1 — Connection handling and crash fixes
557
+
558
+ - Fixed cross-connection crashes across `cleaning_ops.py`,
559
+ `cleaning_ops_extended.py`, and `cleaning_ops_final.py` by registering
560
+ relations on the caller's DuckDB connection instead of creating new
561
+ in-memory connections or relying on `FROM relation` replacement scans.
562
+ - `DuckJanitor.__init__` now validates that the relation and connection
563
+ belong to the same DuckDB connection.
564
+ - `from_parquet`, `from_csv`, and `from_sql` now return real DuckDB
565
+ relations without round-tripping through pandas.
566
+ - Fixed `remove_empty` to actually remove all-empty rows (in addition to
567
+ all-empty columns).
568
+ - Fixed `dropna(how='all')` boolean condition.
569
+ - Fixed `case_when`, `currency_column_to_numeric`, `convert_date`
570
+ `relation.database` AttributeError crashes.
571
+ - Fixed `impute()` `SELECT , COALESCE(...)` syntax error.
572
+ - Fixed `conditional_join` to use a single shared connection with an
573
+ operator allow-list.
574
+ - Replaced invalid `ROW() OVER ()` in `select_rows` with
575
+ `ROW_NUMBER() OVER ()`.
576
+ - Added safer handling for identical-value columns in `min_max_scale`.
577
+ - Added 10 regression tests. Full suite: 40 passing.
578
+
579
+ ### 0.1.0 — Initial release
580
+
581
+ - DuckDB-backed pyjanitor-style cleaning API with 51 functions.
582
+ - Lazy SQL evaluation for simple operations; hybrid SQL/Python for
583
+ complex operations.
584
+
585
+ For a complete release history, see [CHANGELOG.md](CHANGELOG.md).
586
+
587
+ ## License
588
+
589
+ MIT License - see [LICENSE](LICENSE) for details.
590
+
591
+ ## Acknowledgments
592
+
593
+ - [pyjanitor](https://pyjanitor-devs.github.io/pyjanitor/) - Original data cleaning API
594
+ - [DuckDB](https://duckdb.org/) - High-performance analytical database
595
+ - [infer](https://infer.netlify.app/) - Inspiration for the tidy grammar approach
596
+ - [duckplyr](https://duckplyr.tidyverse.org/) - Inspiration for DuckDB-backed tidyverse