freshdata-cleaner 0.1.0__tar.gz

freshdata_cleaner-0.1.0/.gitignore

@@ -0,0 +1,16 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+.venv/
+venv/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+coverage.xml
+htmlcov/
+.DS_Store
+.ipynb_checkpoints/

freshdata_cleaner-0.1.0/CHANGELOG.md

@@ -0,0 +1,28 @@
+# Changelog
+
+All notable changes to this project are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project
+adheres to [Semantic Versioning](https://semver.org/).
+
+## [0.1.0] - 2026-06-12
+
+Initial release.
+
+### Added
+- `freshdata.clean()` — automatic, audited cleaning: column-name
+  normalization, whitespace stripping, sentinel-string normalization,
+  empty row/column pruning, validated dtype inference (numeric incl.
+  currency/thousands separators, datetime, boolean), and exact duplicate
+  removal.
+- Opt-in steps: imputation (`auto`/`mean`/`median`/`mode`), outlier
+  clipping/flagging (IQR or z-score), constant-column dropping, memory
+  optimization (numeric downcasting + category conversion), index reset.
+- `freshdata.profile()` — read-only profiling whose dtype suggestions are
+  produced by the same inference code `clean` uses.
+- `freshdata.Cleaner` — reusable configured pipeline with `report_`.
+- `freshdata.CleanConfig` — frozen, self-validating configuration;
+  unknown options raise with a "did you mean" suggestion.
+- `freshdata.CleanReport` / `freshdata.Action` — structured audit trail
+  with `summary()`, `to_dict()`, `to_frame()`.
+- Type hints throughout (`py.typed`), zero dependencies beyond
+  pandas/numpy, support for Python 3.9–3.13.

freshdata_cleaner-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Johnny Wilson Dougherty
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

freshdata_cleaner-0.1.0/PKG-INFO

@@ -0,0 +1,205 @@
+Metadata-Version: 2.4
+Name: freshdata-cleaner
+Version: 0.1.0
+Summary: Fast, safe, automatic data cleaning for real-world tabular data.
+Project-URL: Homepage, https://github.com/JohnnyWilson-Portfolio/freshdata
+Project-URL: Repository, https://github.com/JohnnyWilson-Portfolio/freshdata
+Project-URL: Issues, https://github.com/JohnnyWilson-Portfolio/freshdata/issues
+Project-URL: Changelog, https://github.com/JohnnyWilson-Portfolio/freshdata/blob/main/CHANGELOG.md
+Author-email: Johnny Wilson Dougherty <jyothiswaroop2803@gmail.com>
+License: MIT
+License-File: LICENSE
+Keywords: data-cleaning,data-quality,etl,pandas,preprocessing,tabular-data
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Requires-Dist: numpy>=1.21
+Requires-Dist: pandas>=1.5
+Provides-Extra: dev
+Requires-Dist: build>=1.0; extra == 'dev'
+Requires-Dist: mypy>=1.8; extra == 'dev'
+Requires-Dist: pytest-cov>=4.0; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: ruff>=0.4; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# freshdata
+
+**Fast, safe, automatic data cleaning for real-world tabular data.**
+
+[![CI](https://github.com/JohnnyWilson-Portfolio/freshdata/actions/workflows/ci.yml/badge.svg)](https://github.com/JohnnyWilson-Portfolio/freshdata/actions/workflows/ci.yml)
+[![Python](https://img.shields.io/badge/python-3.9%2B-blue)](https://pypi.org/project/freshdata-cleaner/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+
+`freshdata` fixes the messy parts of CSV / Excel / SQL-export data — stray
+whitespace, `"N/A"` strings, numbers stored as text, duplicate rows — in one
+call, and tells you exactly what it did.
+
+```python
+import pandas as pd
+import freshdata as fd
+
+df = pd.read_csv("export.csv")
+
+cleaned = fd.clean(df)                      # one line
+cleaned, report = fd.clean(df, report=True) # ... with a full audit trail
+print(report.summary())
+```
+
+```text
+freshdata clean report
+  rows:    5 -> 4 (-1)
+  columns: 6 -> 5 (-1)
+  memory:  1.5 KB -> 298 B
+  time:    0.011s
+  actions (12):
+    - [column_names] renamed 5 column(s): ' First Name '->'first_name', 'AGE'->'age', …
+    - [strip_whitespace] 'first_name': trimmed surrounding whitespace
+    - [normalize_sentinels] 'age': replaced sentinel strings ("N/A", "-", "", …) with missing
+    - [drop_empty_columns] dropped 1 all-missing column(s): empty
+    - [fix_dtypes] 'age': converted to Int64
+    - [fix_dtypes] 'joined_date': converted to datetime64[ns]
+    - [fix_dtypes] 'active': converted to bool
+    - [fix_dtypes] 'salary': converted to float64
+    - [drop_duplicates] dropped 1 duplicate row(s)
+```
+
+## Install
+
+```bash
+pip install freshdata-cleaner
+```
+
+Requires Python ≥ 3.9 and pandas ≥ 1.5. No other dependencies.
+
+## Why another cleaning library?
+
+Most auto-cleaners are either trivial wrappers or opaque frameworks that
+guess. `freshdata` is built on four rules:
+
+1. **No surprises.** Defaults only repair *representation* — whitespace,
+   sentinel strings, wrong dtypes, exact duplicate rows, all-empty
+   rows/columns. Anything that changes your data's *statistics* (imputation,
+   outlier handling, lossy downcasting) is opt-in.
+2. **Everything is reported.** Every transformation is recorded with the
+   column name and the number of affected cells. `bool(report)` is `False`
+   when nothing changed.
+3. **Never mutates your input.** `clean` returns a new frame (built from a
+   shallow copy, so unchanged columns cost no extra memory). `profile` is
+   read-only.
+4. **Fast by construction.** Vectorized pandas operations only — no
+   row-wise `apply`. Type inference pre-screens a sample, so hopeless
+   conversions are rejected at O(sample), not O(n), and conversions only
+   stick when ≥ 95 % of values parse (configurable).
+
+## What `clean` does by default
+
+| order | step | what it does |
+|---|---|---|
+| 1 | `column_names` | snake_case names, deduplicate collisions (`"a", "a"` → `"a", "a_2"`) |
+| 2 | `strip_whitespace` | trim surrounding whitespace in text cells (internal spacing kept) |
+| 3 | `normalize_sentinels` | `"N/A"`, `"null"`, `"-"`, `""`, `"#REF!"`, … → missing |
+| 4 | `drop_empty_columns` / `drop_empty_rows` | remove all-missing columns and rows |
+| 5 | `fix_dtypes` | text → numeric (`"$1,234.56"` works) / datetime / boolean, validated |
+| 6 | `drop_duplicates` | drop exact duplicate rows, keep the first |
+
+Conversions are conservative: a column converts only when at least
+`numeric_threshold` (default 0.95) of its non-missing values parse, mixed-type
+columns never lose their non-string values, and every value coerced to missing
+is counted in the report.
+
+## Opt-in steps
+
+```python
+fd.clean(
+    df,
+    impute="auto",              # median for numeric, mode otherwise ("mean"/"median"/"mode")
+    outliers="clip",            # or "flag" to add a boolean <col>_outlier column
+    outlier_method="iqr",       # or "zscore"; factors default to 1.5 / 3.0
+    drop_constant_columns=True, # single-valued columns
+    optimize_memory=True,       # downcast numerics, categorize low-cardinality text
+    reset_index=True,           # 0..n-1 index instead of original labels
+)
+```
+
+Every option lives on one frozen dataclass — `fd.CleanConfig` — and unknown
+names fail immediately with a "did you mean" suggestion:
+
+```python
+config = fd.CleanConfig(drop_duplicates=False, extra_sentinels=("unknown",))
+fd.clean(df, config=config, impute="median")   # config + overrides
+
+cleaner = fd.Cleaner(impute="median")          # reusable pipeline
+for path in paths:
+    out = cleaner.clean(pd.read_csv(path))
+    log.info(cleaner.report_.summary())
+```
+
+## Profiling
+
+`fd.profile(df)` inspects without changing anything — and because it runs the
+*same* inference code as `clean`, its suggestions are a faithful preview:
+
+```python
+print(fd.profile(df))
+```
+
+```text
+freshdata profile — 5 rows x 6 columns, 1.5 KB
+  missing cells: 6 (20.0%)   duplicate rows: 1
+  column        dtype    missing  issues
+   First Name   object       20%  20.0% missing; 1 value(s) with surrounding whitespace; …
+  AGE           object         -  1 sentinel value(s) meaning missing; would convert to Int64
+  Joined Date   object         -  would convert to datetime64[ns]
+  Active        object         -  would convert to bool
+  Salary($)     object         -  would convert to float64
+  empty         object      100%  100.0% missing; constant column
+```
+
+`profile.to_frame()` gives the same as a DataFrame; `profile.to_dict()` is
+JSON-friendly for logging and data-quality dashboards.
+
+## What freshdata will not do
+
+- Guess at fuzzy entity resolution ("Jon" vs "John").
+- Impute, drop outliers, or change distributions unless you ask.
+- Parse ambiguous European decimal commas (`"1.234,56"`) — too risky to guess.
+- Mutate your DataFrame, ever.
+
+## API
+
+| name | purpose |
+|---|---|
+| `fd.clean(df, *, report=False, config=None, **options)` | clean, optionally returning a `CleanReport` |
+| `fd.profile(df, *, config=None, **options)` | read-only inspection with actionable issues |
+| `fd.Cleaner(config=None, **options)` | reusable configured pipeline (`.clean()`, `.report_`) |
+| `fd.CleanConfig` | frozen dataclass holding every option |
+| `fd.CleanReport` / `fd.Action` | audit trail (`summary()`, `to_dict()`, `to_frame()`) |
+| `fd.Profile` / `fd.ColumnProfile` | profiling results |
+
+## Development
+
+```bash
+git clone https://github.com/JohnnyWilson-Portfolio/freshdata
+cd freshdata
+pip install -e ".[dev]"
+pytest
+ruff check src tests
+```
+
+Benchmarks live in `benchmarks/bench.py` (`python benchmarks/bench.py`).
+
+## License
+
+MIT — see [LICENSE](LICENSE).

freshdata_cleaner-0.1.0/README.md

@@ -0,0 +1,169 @@
+# freshdata
+
+**Fast, safe, automatic data cleaning for real-world tabular data.**
+
+[![CI](https://github.com/JohnnyWilson-Portfolio/freshdata/actions/workflows/ci.yml/badge.svg)](https://github.com/JohnnyWilson-Portfolio/freshdata/actions/workflows/ci.yml)
+[![Python](https://img.shields.io/badge/python-3.9%2B-blue)](https://pypi.org/project/freshdata-cleaner/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+
+`freshdata` fixes the messy parts of CSV / Excel / SQL-export data — stray
+whitespace, `"N/A"` strings, numbers stored as text, duplicate rows — in one
+call, and tells you exactly what it did.
+
+```python
+import pandas as pd
+import freshdata as fd
+
+df = pd.read_csv("export.csv")
+
+cleaned = fd.clean(df)                      # one line
+cleaned, report = fd.clean(df, report=True) # ... with a full audit trail
+print(report.summary())
+```
+
+```text
+freshdata clean report
+  rows:    5 -> 4 (-1)
+  columns: 6 -> 5 (-1)
+  memory:  1.5 KB -> 298 B
+  time:    0.011s
+  actions (12):
+    - [column_names] renamed 5 column(s): ' First Name '->'first_name', 'AGE'->'age', …
+    - [strip_whitespace] 'first_name': trimmed surrounding whitespace
+    - [normalize_sentinels] 'age': replaced sentinel strings ("N/A", "-", "", …) with missing
+    - [drop_empty_columns] dropped 1 all-missing column(s): empty
+    - [fix_dtypes] 'age': converted to Int64
+    - [fix_dtypes] 'joined_date': converted to datetime64[ns]
+    - [fix_dtypes] 'active': converted to bool
+    - [fix_dtypes] 'salary': converted to float64
+    - [drop_duplicates] dropped 1 duplicate row(s)
+```
+
+## Install
+
+```bash
+pip install freshdata-cleaner
+```
+
+Requires Python ≥ 3.9 and pandas ≥ 1.5. No other dependencies.
+
+## Why another cleaning library?
+
+Most auto-cleaners are either trivial wrappers or opaque frameworks that
+guess. `freshdata` is built on four rules:
+
+1. **No surprises.** Defaults only repair *representation* — whitespace,
+   sentinel strings, wrong dtypes, exact duplicate rows, all-empty
+   rows/columns. Anything that changes your data's *statistics* (imputation,
+   outlier handling, lossy downcasting) is opt-in.
+2. **Everything is reported.** Every transformation is recorded with the
+   column name and the number of affected cells. `bool(report)` is `False`
+   when nothing changed.
+3. **Never mutates your input.** `clean` returns a new frame (built from a
+   shallow copy, so unchanged columns cost no extra memory). `profile` is
+   read-only.
+4. **Fast by construction.** Vectorized pandas operations only — no
+   row-wise `apply`. Type inference pre-screens a sample, so hopeless
+   conversions are rejected at O(sample), not O(n), and conversions only
+   stick when ≥ 95 % of values parse (configurable).
+
+## What `clean` does by default
+
+| order | step | what it does |
+|---|---|---|
+| 1 | `column_names` | snake_case names, deduplicate collisions (`"a", "a"` → `"a", "a_2"`) |
+| 2 | `strip_whitespace` | trim surrounding whitespace in text cells (internal spacing kept) |
+| 3 | `normalize_sentinels` | `"N/A"`, `"null"`, `"-"`, `""`, `"#REF!"`, … → missing |
+| 4 | `drop_empty_columns` / `drop_empty_rows` | remove all-missing columns and rows |
+| 5 | `fix_dtypes` | text → numeric (`"$1,234.56"` works) / datetime / boolean, validated |
+| 6 | `drop_duplicates` | drop exact duplicate rows, keep the first |
+
+Conversions are conservative: a column converts only when at least
+`numeric_threshold` (default 0.95) of its non-missing values parse, mixed-type
+columns never lose their non-string values, and every value coerced to missing
+is counted in the report.
+
+## Opt-in steps
+
+```python
+fd.clean(
+    df,
+    impute="auto",              # median for numeric, mode otherwise ("mean"/"median"/"mode")
+    outliers="clip",            # or "flag" to add a boolean <col>_outlier column
+    outlier_method="iqr",       # or "zscore"; factors default to 1.5 / 3.0
+    drop_constant_columns=True, # single-valued columns
+    optimize_memory=True,       # downcast numerics, categorize low-cardinality text
+    reset_index=True,           # 0..n-1 index instead of original labels
+)
+```
+
+Every option lives on one frozen dataclass — `fd.CleanConfig` — and unknown
+names fail immediately with a "did you mean" suggestion:
+
+```python
+config = fd.CleanConfig(drop_duplicates=False, extra_sentinels=("unknown",))
+fd.clean(df, config=config, impute="median")   # config + overrides
+
+cleaner = fd.Cleaner(impute="median")          # reusable pipeline
+for path in paths:
+    out = cleaner.clean(pd.read_csv(path))
+    log.info(cleaner.report_.summary())
+```
+
+## Profiling
+
+`fd.profile(df)` inspects without changing anything — and because it runs the
+*same* inference code as `clean`, its suggestions are a faithful preview:
+
+```python
+print(fd.profile(df))
+```
+
+```text
+freshdata profile — 5 rows x 6 columns, 1.5 KB
+  missing cells: 6 (20.0%)   duplicate rows: 1
+  column        dtype    missing  issues
+   First Name   object       20%  20.0% missing; 1 value(s) with surrounding whitespace; …
+  AGE           object         -  1 sentinel value(s) meaning missing; would convert to Int64
+  Joined Date   object         -  would convert to datetime64[ns]
+  Active        object         -  would convert to bool
+  Salary($)     object         -  would convert to float64
+  empty         object      100%  100.0% missing; constant column
+```
+
+`profile.to_frame()` gives the same as a DataFrame; `profile.to_dict()` is
+JSON-friendly for logging and data-quality dashboards.
+
+## What freshdata will not do
+
+- Guess at fuzzy entity resolution ("Jon" vs "John").
+- Impute, drop outliers, or change distributions unless you ask.
+- Parse ambiguous European decimal commas (`"1.234,56"`) — too risky to guess.
+- Mutate your DataFrame, ever.
+
+## API
+
+| name | purpose |
+|---|---|
+| `fd.clean(df, *, report=False, config=None, **options)` | clean, optionally returning a `CleanReport` |
+| `fd.profile(df, *, config=None, **options)` | read-only inspection with actionable issues |
+| `fd.Cleaner(config=None, **options)` | reusable configured pipeline (`.clean()`, `.report_`) |
+| `fd.CleanConfig` | frozen dataclass holding every option |
+| `fd.CleanReport` / `fd.Action` | audit trail (`summary()`, `to_dict()`, `to_frame()`) |
+| `fd.Profile` / `fd.ColumnProfile` | profiling results |
+
+## Development
+
+```bash
+git clone https://github.com/JohnnyWilson-Portfolio/freshdata
+cd freshdata
+pip install -e ".[dev]"
+pytest
+ruff check src tests
+```
+
+Benchmarks live in `benchmarks/bench.py` (`python benchmarks/bench.py`).
+
+## License
+
+MIT — see [LICENSE](LICENSE).

freshdata_cleaner-0.1.0/pyproject.toml

@@ -0,0 +1,96 @@
+[build-system]
+requires = ["hatchling>=1.21"]
+build-backend = "hatchling.build"
+
+[project]
+name = "freshdata-cleaner"
+version = "0.1.0"
+description = "Fast, safe, automatic data cleaning for real-world tabular data."
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "MIT" }
+authors = [
+    { name = "Johnny Wilson Dougherty", email = "jyothiswaroop2803@gmail.com" },
+]
+keywords = [
+    "data-cleaning",
+    "data-quality",
+    "pandas",
+    "preprocessing",
+    "etl",
+    "tabular-data",
+]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Information Analysis",
+    "Typing :: Typed",
+]
+dependencies = [
+    "pandas>=1.5",
+    "numpy>=1.21",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0",
+    "pytest-cov>=4.0",
+    "ruff>=0.4",
+    "mypy>=1.8",
+    "build>=1.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/JohnnyWilson-Portfolio/freshdata"
+Repository = "https://github.com/JohnnyWilson-Portfolio/freshdata"
+Issues = "https://github.com/JohnnyWilson-Portfolio/freshdata/issues"
+Changelog = "https://github.com/JohnnyWilson-Portfolio/freshdata/blob/main/CHANGELOG.md"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/freshdata"]
+
+[tool.hatch.build.targets.sdist]
+include = ["src", "tests", "README.md", "CHANGELOG.md", "LICENSE"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-q --strict-markers"
+xfail_strict = true
+
+[tool.ruff]
+line-length = 99
+target-version = "py39"
+src = ["src", "tests"]
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "UP", "B", "SIM", "C4", "RET", "PL"]
+ignore = [
+    "PLR0911", # too many returns — dtype heuristics are naturally branchy
+    "PLR0912", # too many branches — cleaning steps are naturally branchy
+    "PLR0913", # many keyword options is the point of the config surface
+    "PLR2004", # magic-value comparisons in heuristics are documented inline
+]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/*" = ["PLR2004", "SIM117"]
+
+[tool.mypy]
+python_version = "3.9"
+strict = false
+warn_unused_ignores = true
+warn_redundant_casts = true
+no_implicit_optional = true
+files = ["src/freshdata"]
+
+[[tool.mypy.overrides]]
+module = ["pandas.*", "numpy.*"]
+ignore_missing_imports = true

freshdata_cleaner-0.1.0/src/freshdata/__init__.py

@@ -0,0 +1,39 @@
+"""freshdata — fast, safe, automatic data cleaning for real-world tabular data.
+
+>>> import freshdata as fd
+>>> cleaned = fd.clean(df)
+>>> cleaned, report = fd.clean(df, report=True)
+>>> print(fd.profile(df))
+
+Design principles
+-----------------
+- **No surprises.** Defaults only fix representation (whitespace, sentinel
+  strings, wrong dtypes, exact duplicates, empty rows/columns). Anything that
+  changes your data's statistics is opt-in.
+- **Everything is reported.** Each transformation is recorded with the column
+  and the number of affected cells.
+- **Never mutates input.** ``clean`` returns a new frame; profiling is
+  read-only.
+- **Fast by construction.** Vectorized pandas operations only, with
+  sample-based pre-screening so type inference stays cheap on large frames.
+"""
+
+from .api import clean, profile
+from .cleaner import Cleaner
+from .config import CleanConfig
+from .profile import ColumnProfile, Profile
+from .report import Action, CleanReport
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "Action",
+    "CleanConfig",
+    "CleanReport",
+    "Cleaner",
+    "ColumnProfile",
+    "Profile",
+    "__version__",
+    "clean",
+    "profile",
+]

freshdata_cleaner-0.1.0/src/freshdata/_sentinels.py

@@ -0,0 +1,50 @@
+"""Registry of string values that conventionally mean "missing".
+
+All entries are stored casefolded; matching is case-insensitive and happens
+after whitespace stripping, so ``" N/A "`` and ``"n/a"`` both match.
+"""
+
+from __future__ import annotations
+
+#: Values commonly used in CSV / Excel / SQL exports to denote a missing cell.
+#: Deliberately conservative: entries here are near-certain to mean "missing"
+#: when they appear as the entire cell value. Domain words that merely *might*
+#: mean missing (e.g. ``"unknown"``) are excluded; pass them via the
+#: ``extra_sentinels`` option instead.
+DEFAULT_SENTINELS: frozenset[str] = frozenset(
+    {
+        # empty / placeholder punctuation
+        "",
+        "-",
+        "--",
+        "---",
+        "?",
+        "??",
+        # spelled-out missing markers
+        "na",
+        "n/a",
+        "n\\a",
+        "n.a",
+        "n.a.",
+        "nan",
+        "null",
+        "none",
+        "nil",
+        "missing",
+        "(null)",
+        "(none)",
+        "(blank)",
+        "(empty)",
+        "(missing)",
+        # Excel error codes — never legitimate data
+        "#n/a",
+        "#n/a n/a",
+        "#na",
+        "#null!",
+        "#div/0!",
+        "#ref!",
+        "#value!",
+        "#name?",
+        "#num!",
+    }
+)

freshdata_cleaner-0.1.0/src/freshdata/_util.py

@@ -0,0 +1,59 @@
+"""Small shared helpers. Internal — no stability guarantees."""
+
+from __future__ import annotations
+
+import pandas as pd
+
+#: Major version of the installed pandas, for the few places behavior differs.
+PANDAS_MAJOR: int = int(pd.__version__.split(".")[0])
+
+
+#: Above this many rows, object payloads are estimated from a sample instead
+#: of measured cell by cell, keeping report bookkeeping ~free on tall frames.
+_MEMORY_SAMPLE_THRESHOLD = 200_000
+_MEMORY_SAMPLE_SIZE = 20_000
+
+
+def memory_bytes(df: pd.DataFrame) -> int:
+    """Total memory footprint of *df* in bytes, including object payloads.
+
+    Exact for frames up to ~200k rows; for taller frames the per-row payload
+    of object/string columns is estimated from a 20k-row random sample (other
+    dtypes are always exact — their size does not depend on values).
+    """
+    n = len(df)
+    if n <= _MEMORY_SAMPLE_THRESHOLD:
+        return int(df.memory_usage(deep=True).sum())
+    total = int(df.memory_usage(deep=False).sum())
+    for i, dtype in enumerate(df.dtypes):
+        if not _is_stringlike_dtype(dtype):
+            continue
+        sample = df.iloc[:, i].sample(_MEMORY_SAMPLE_SIZE, random_state=0)
+        payload = sample.memory_usage(deep=True) - sample.memory_usage(deep=False)
+        total += int(payload / len(sample) * n)
+    return total
+
+
+def format_bytes(n: float) -> str:
+    """Render a byte count for humans: ``format_bytes(2048) == '2.0 KB'``."""
+    for unit in ("B", "KB", "MB", "GB"):
+        if abs(n) < 1024.0:
+            return f"{n:.1f} {unit}" if unit != "B" else f"{int(n)} B"
+        n /= 1024.0
+    return f"{n:.1f} TB"
+
+
+def sample_series(s: pd.Series, size: int, random_state: int) -> pd.Series:
+    """Return *s* itself if small, else a reproducible random sample of *size*."""
+    if len(s) <= size:
+        return s
+    return s.sample(size, random_state=random_state)
+
+
+def stringlike_columns(df: pd.DataFrame) -> list:
+    """Column labels whose dtype can hold free-form text (object or string)."""
+    return list(df.columns[[_is_stringlike_dtype(dt) for dt in df.dtypes]])
+
+
+def _is_stringlike_dtype(dtype: object) -> bool:
+    return pd.api.types.is_object_dtype(dtype) or isinstance(dtype, pd.StringDtype)