unitsleuth 0.1.0__tar.gz

unitsleuth-0.1.0/.gitignore

@@ -0,0 +1,10 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv

unitsleuth-0.1.0/CHANGELOG.md

@@ -0,0 +1,9 @@
+# Changelog
+
+## 0.1.0
+
+- Catalog of affine unit conversions (length, mass, temperature, volume, pressure, speed, time, ratios, power-of-ten errors).
+- Univariate mixture detector with per-row posteriors and BIC-based evidence.
+- Row-consistent multivariate detection and repair.
+- pandas and Polars support through Narwhals.
+- Command-line interface.

unitsleuth-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Md Omar Faruqe
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

unitsleuth-0.1.0/PKG-INFO

@@ -0,0 +1,88 @@
+Metadata-Version: 2.5
+Name: unitsleuth
+Version: 0.1.0
+Summary: Detect mixed measurement units in unlabeled numeric columns from the numbers alone.
+Project-URL: Homepage, https://github.com/omarfaruqe/unitsleuth
+Project-URL: Issues, https://github.com/omarfaruqe/unitsleuth/issues
+Author-email: Md Omar Faruqe <faruqe@ru.ac.bd>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: data-cleaning,data-quality,measurement,mixture-model,units
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: narwhals>=1.20
+Requires-Dist: numpy>=1.24
+Description-Content-Type: text/markdown
+
+# unitsleuth
+
+Detect mixed measurement units in an unlabeled numeric column, from the numbers alone.
+
+A height column where most rows are centimetres and a few are inches, or a
+temperature column that mixes °C and °F, looks fine to a schema check. unitsleuth
+fits a mixture model per known unit conversion and tells you which rows were
+probably recorded in the other unit.
+
+This is a Python implementation of the mixture-model approach of Di Zio, Guarnera
+and Luzi (2005, *Survey Methodology*), extended with affine conversions
+(e.g. °C to °F) and a catalog of common unit pairs. It is not a new method.
+
+## Quick start
+
+```python
+import numpy as np
+import unitsleuth as us
+
+heights = np.concatenate([np.random.normal(170, 9, 900), np.random.normal(67, 3.5, 100)])
+best = us.detect(heights, quantity="length")[0]
+print(best.label, best.weight, best.detected)
+rows = best.converted  # boolean mask of rows likely in the other unit
+```
+
+### Whole rows in the wrong unit
+
+When a record was entered in one unit system, every column in that row is
+affected. `detect_rows` shares one flag per row across columns, and `repair`
+harmonises the data. `df` can be a pandas or Polars DataFrame (via
+[Narwhals](https://narwhals-dev.github.io/narwhals/)) or a dict of arrays:
+
+```python
+cat = us.catalog
+fit = us.detect_rows(df, {"height": cat.get("cm", "in"), "weight": cat.get("kg", "lb")})
+print(fit.summary())
+fixed = us.repair(df, fit)  # same dataframe type, columns corrected
+```
+
+### Command line
+
+```
+unitsleuth data.csv --col height:cm,in --col weight:kg,lb --out data_fixed.csv
+```
+
+Prints the report and, with `--out`, writes a harmonised CSV. The exit code is 1
+when mixed units are detected, 0 when not and 2 on usage errors, so it can gate a
+data pipeline.
+
+## Status
+
+Alpha. Univariate and row-consistent multivariate detection and repair work;
+dataframe integration (pandas, Polars) and a CLI are planned. Detection needs the units to be
+separated relative to the spread of the data, so it cannot distinguish them when
+natural variation is wide compared with the conversion factor.
+
+## Development
+
+```
+uv sync
+uv run pytest
+```
+
+## Validation
+
+- **Negative control (real data):** on the Palmer penguins dataset (342 birds, 4 numeric columns, 24 catalog conversions each), nothing was flagged: 0 of 96 tests, despite three species with different body sizes.
+- **Positive control (semi-real):** converting a known 5%, 15% or 30% of the penguins' body mass (kg to lb) and bill length (cm to in) and running `detect_rows` over 3 random seeds each, the injected rows were found with 100% row accuracy in all 9 runs, and the estimated share was close to the injected share.
+- **Limits:** these results are for well-separated unit pairs. When the natural spread of a column is wide relative to the conversion factor, the units cannot be told apart from the numbers alone, and the evidence score will say so.

unitsleuth-0.1.0/README.md

@@ -0,0 +1,68 @@
+# unitsleuth
+
+Detect mixed measurement units in an unlabeled numeric column, from the numbers alone.
+
+A height column where most rows are centimetres and a few are inches, or a
+temperature column that mixes °C and °F, looks fine to a schema check. unitsleuth
+fits a mixture model per known unit conversion and tells you which rows were
+probably recorded in the other unit.
+
+This is a Python implementation of the mixture-model approach of Di Zio, Guarnera
+and Luzi (2005, *Survey Methodology*), extended with affine conversions
+(e.g. °C to °F) and a catalog of common unit pairs. It is not a new method.
+
+## Quick start
+
+```python
+import numpy as np
+import unitsleuth as us
+
+heights = np.concatenate([np.random.normal(170, 9, 900), np.random.normal(67, 3.5, 100)])
+best = us.detect(heights, quantity="length")[0]
+print(best.label, best.weight, best.detected)
+rows = best.converted  # boolean mask of rows likely in the other unit
+```
+
+### Whole rows in the wrong unit
+
+When a record was entered in one unit system, every column in that row is
+affected. `detect_rows` shares one flag per row across columns, and `repair`
+harmonises the data. `df` can be a pandas or Polars DataFrame (via
+[Narwhals](https://narwhals-dev.github.io/narwhals/)) or a dict of arrays:
+
+```python
+cat = us.catalog
+fit = us.detect_rows(df, {"height": cat.get("cm", "in"), "weight": cat.get("kg", "lb")})
+print(fit.summary())
+fixed = us.repair(df, fit)  # same dataframe type, columns corrected
+```
+
+### Command line
+
+```
+unitsleuth data.csv --col height:cm,in --col weight:kg,lb --out data_fixed.csv
+```
+
+Prints the report and, with `--out`, writes a harmonised CSV. The exit code is 1
+when mixed units are detected, 0 when not and 2 on usage errors, so it can gate a
+data pipeline.
+
+## Status
+
+Alpha. Univariate and row-consistent multivariate detection and repair work;
+dataframe integration (pandas, Polars) and a CLI are planned. Detection needs the units to be
+separated relative to the spread of the data, so it cannot distinguish them when
+natural variation is wide compared with the conversion factor.
+
+## Development
+
+```
+uv sync
+uv run pytest
+```
+
+## Validation
+
+- **Negative control (real data):** on the Palmer penguins dataset (342 birds, 4 numeric columns, 24 catalog conversions each), nothing was flagged: 0 of 96 tests, despite three species with different body sizes.
+- **Positive control (semi-real):** converting a known 5%, 15% or 30% of the penguins' body mass (kg to lb) and bill length (cm to in) and running `detect_rows` over 3 random seeds each, the injected rows were found with 100% row accuracy in all 9 runs, and the estimated share was close to the injected share.
+- **Limits:** these results are for well-separated unit pairs. When the natural spread of a column is wide relative to the conversion factor, the units cannot be told apart from the numbers alone, and the evidence score will say so.

unitsleuth-0.1.0/pyproject.toml

@@ -0,0 +1,61 @@
+[project]
+name = "unitsleuth"
+version = "0.1.0"
+description = "Detect mixed measurement units in unlabeled numeric columns from the numbers alone."
+readme = "README.md"
+license = "MIT"
+authors = [{ name = "Md Omar Faruqe", email = "faruqe@ru.ac.bd" }]
+requires-python = ">=3.10"
+keywords = ["data-quality", "units", "measurement", "mixture-model", "data-cleaning"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Science/Research",
+    "Programming Language :: Python :: 3",
+    "Topic :: Scientific/Engineering :: Information Analysis",
+    "Typing :: Typed",
+]
+dependencies = [
+    "narwhals>=1.20",
+    "numpy>=1.24",
+]
+
+[project.scripts]
+unitsleuth = "unitsleuth.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/omarfaruqe/unitsleuth"
+Issues = "https://github.com/omarfaruqe/unitsleuth/issues"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[dependency-groups]
+dev = [
+    "pytest>=8",
+    "pytest-cov>=5",
+    "hypothesis>=6",
+    "ruff>=0.6",
+    "mypy>=1.11",
+    "pandas>=2.3.3",
+    "polars>=2.0.0",
+    "pyarrow>=25.0.1",
+]
+
+[tool.ruff]
+line-length = 88
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B", "SIM", "NPY"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-ra --strict-markers"
+
+[tool.mypy]
+strict = true
+python_version = "3.10"
+
+[tool.hatch.build.targets.sdist]
+include = ["src", "tests", "README.md", "LICENSE", "CHANGELOG.md", "pyproject.toml"]

unitsleuth-0.1.0/src/unitsleuth/__init__.py

@@ -0,0 +1,19 @@
+"""unitsleuth: detect mixed measurement units in numeric columns."""
+
+from unitsleuth import catalog
+from unitsleuth.catalog import Conversion
+from unitsleuth.detect import UnivariateFit, detect, fit_conversion
+from unitsleuth.multivariate import RowFit, detect_rows, repair
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "Conversion",
+    "RowFit",
+    "UnivariateFit",
+    "catalog",
+    "detect",
+    "detect_rows",
+    "fit_conversion",
+    "repair",
+]

unitsleuth-0.1.0/src/unitsleuth/__main__.py

@@ -0,0 +1,3 @@
+from unitsleuth.cli import main
+
+raise SystemExit(main())

unitsleuth-0.1.0/src/unitsleuth/_frame.py

@@ -0,0 +1,50 @@
+"""Dataframe adapters built on Narwhals (pandas, Polars, PyArrow, ...)."""
+
+from __future__ import annotations
+
+from collections.abc import Mapping
+from typing import Any
+
+import narwhals.stable.v1 as nw
+import numpy as np
+from numpy.typing import ArrayLike, NDArray
+
+FloatArray = NDArray[np.float64]
+
+
+def is_frame(obj: Any) -> bool:
+    """True for native eager dataframes that Narwhals understands."""
+    if isinstance(obj, Mapping):
+        return False
+    try:
+        nw.from_native(obj, eager_only=True)
+    except TypeError:
+        return False
+    return True
+
+
+def columns_of(data: Any, names: list[str] | tuple[str, ...]) -> dict[str, ArrayLike]:
+    """Extract named columns as float arrays from a mapping or a dataframe."""
+    if isinstance(data, Mapping):
+        return {n: data[n] for n in names}
+    frame = nw.from_native(data, eager_only=True)
+    missing = [n for n in names if n not in frame.columns]
+    if missing:
+        raise KeyError(f"columns not in data: {missing}")
+    out: dict[str, ArrayLike] = {}
+    for n in names:
+        s = frame[n].cast(nw.Float64)
+        out[n] = np.array(s.to_list(), dtype=np.float64)
+        # to_list maps nulls to None; np.array(..., float64) turns None into nan
+    return out
+
+
+def replace_columns(data: Any, cols: Mapping[str, FloatArray]) -> Any:
+    """Return a native dataframe of the same type with ``cols`` replaced."""
+    frame = nw.from_native(data, eager_only=True)
+    ns = nw.get_native_namespace(frame)
+    new = [
+        nw.new_series(name, arr.tolist(), nw.Float64, native_namespace=ns)
+        for name, arr in cols.items()
+    ]
+    return frame.with_columns(*new).to_native()

unitsleuth-0.1.0/src/unitsleuth/catalog.py

@@ -0,0 +1,117 @@
+"""Catalog of unit conversions that unitsleuth can test for.
+
+Every conversion is affine: ``x_to = scale * x_from + offset``. Multiplicative
+conversions (``offset == 0``) cover most unit pairs; temperature needs the offset.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+import numpy as np
+from numpy.typing import ArrayLike, NDArray
+
+FloatArray = NDArray[np.float64]
+
+
+@dataclass(frozen=True)
+class Conversion:
+    """An affine map between two units of the same quantity."""
+
+    quantity: str
+    from_unit: str
+    to_unit: str
+    scale: float
+    offset: float = 0.0
+
+    def __post_init__(self) -> None:
+        if not (np.isfinite(self.scale) and np.isfinite(self.offset)):
+            raise ValueError("scale and offset must be finite")
+        if self.scale == 0.0:
+            raise ValueError("scale must be non-zero")
+
+    @property
+    def name(self) -> str:
+        """Short label such as ``'kg->lb'``."""
+        return f"{self.from_unit}->{self.to_unit}"
+
+    @property
+    def is_multiplicative(self) -> bool:
+        """True when the conversion has no additive offset."""
+        return self.offset == 0.0
+
+    def forward(self, x: ArrayLike) -> FloatArray:
+        """Convert values expressed in ``from_unit`` to ``to_unit``."""
+        arr = np.asarray(x, dtype=np.float64)
+        return np.asarray(self.scale * arr + self.offset, dtype=np.float64)
+
+    def inverse(self, y: ArrayLike) -> FloatArray:
+        """Convert values expressed in ``to_unit`` back to ``from_unit``."""
+        arr = np.asarray(y, dtype=np.float64)
+        return np.asarray((arr - self.offset) / self.scale, dtype=np.float64)
+
+
+def _c(
+    quantity: str, from_unit: str, to_unit: str, scale: float, offset: float = 0.0
+) -> Conversion:
+    return Conversion(quantity, from_unit, to_unit, scale, offset)
+
+
+_CATALOG: tuple[Conversion, ...] = (
+    # length
+    _c("length", "m", "ft", 3.280839895013123),
+    _c("length", "cm", "in", 1.0 / 2.54),
+    _c("length", "km", "mi", 0.621371192237334),
+    _c("length", "m", "cm", 100.0),
+    _c("length", "ft", "in", 12.0),
+    # mass
+    _c("mass", "kg", "lb", 2.2046226218487757),
+    _c("mass", "kg", "g", 1000.0),
+    _c("mass", "lb", "oz", 16.0),
+    _c("mass", "g", "oz", 0.035273961949580414),
+    # temperature
+    _c("temperature", "degC", "degF", 1.8, 32.0),
+    _c("temperature", "degC", "K", 1.0, 273.15),
+    # volume
+    _c("volume", "L", "mL", 1000.0),
+    _c("volume", "L", "gal_us", 0.26417205235814845),
+    # pressure
+    _c("pressure", "kPa", "Pa", 1000.0),
+    _c("pressure", "kPa", "psi", 0.14503773773375),
+    _c("pressure", "mmHg", "kPa", 0.133322387415),
+    # speed
+    _c("speed", "km/h", "mph", 0.621371192237334),
+    _c("speed", "km/h", "m/s", 1.0 / 3.6),
+    # time
+    _c("time", "h", "min", 60.0),
+    _c("time", "min", "s", 60.0),
+    # proportions
+    _c("ratio", "fraction", "percent", 100.0),
+    # generic power-of-ten scale errors (classic keying/recording mistakes)
+    _c("generic", "x1", "x10", 10.0),
+    _c("generic", "x1", "x100", 100.0),
+    _c("generic", "x1", "x1000", 1000.0),
+)
+
+
+def conversions(quantity: str | None = None) -> tuple[Conversion, ...]:
+    """Return the built-in conversions, optionally filtered by quantity."""
+    if quantity is None:
+        return _CATALOG
+    selected = tuple(c for c in _CATALOG if c.quantity == quantity)
+    if not selected:
+        raise KeyError(f"unknown quantity {quantity!r}; known: {quantities()}")
+    return selected
+
+
+def quantities() -> tuple[str, ...]:
+    """Return the quantities covered by the catalog, in catalog order."""
+    return tuple(dict.fromkeys(c.quantity for c in _CATALOG))
+
+
+def get(from_unit: str, to_unit: str) -> Conversion:
+    """Look up a built-in conversion by its unit names."""
+    for conv in _CATALOG:
+        if conv.from_unit == from_unit and conv.to_unit == to_unit:
+            return conv
+    raise KeyError(f"no built-in conversion {from_unit}->{to_unit}")

unitsleuth-0.1.0/src/unitsleuth/cli.py

@@ -0,0 +1,104 @@
+"""Command-line interface: ``unitsleuth data.csv --col height:cm,in``."""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import sys
+from collections.abc import Sequence
+
+import numpy as np
+
+from unitsleuth import catalog
+from unitsleuth.catalog import Conversion
+from unitsleuth.multivariate import detect_rows, repair
+
+
+def _resolve(spec: str) -> tuple[str, Conversion]:
+    try:
+        name, units = spec.rsplit(":", 1)
+        a, b = (u.strip() for u in units.split(","))
+    except ValueError:
+        raise argparse.ArgumentTypeError(
+            f"bad --col {spec!r}; expected NAME:FROM,TO such as height:cm,in"
+        ) from None
+    try:
+        return name, catalog.get(a, b)
+    except KeyError:
+        pass
+    try:
+        c = catalog.get(b, a)
+    except KeyError:
+        raise argparse.ArgumentTypeError(
+            f"no built-in conversion between {a!r} and {b!r}"
+        ) from None
+    return name, Conversion(c.quantity, a, b, 1.0 / c.scale, -c.offset / c.scale)
+
+
+def _num(s: str) -> float:
+    try:
+        return float(s)
+    except ValueError:
+        return float("nan")
+
+
+def build_parser() -> argparse.ArgumentParser:
+    p = argparse.ArgumentParser(
+        prog="unitsleuth",
+        description="Detect rows recorded in a different measurement unit.",
+    )
+    p.add_argument("csv", help="input CSV file with a header row")
+    p.add_argument(
+        "--col",
+        action="append",
+        required=True,
+        type=_resolve,
+        metavar="NAME:FROM,TO",
+        help="column and the conversion to test, e.g. weight:kg,lb (repeatable)",
+    )
+    p.add_argument("--out", help="write a repaired CSV to this path")
+    p.add_argument(
+        "--target",
+        choices=["majority", "base", "converted"],
+        default="majority",
+        help="unit to harmonise to when repairing (default: majority)",
+    )
+    p.add_argument("--min-delta-bic", type=float, default=10.0)
+    return p
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+    """Run the CLI. Exit code 1 means mixed units were detected."""
+    args = build_parser().parse_args(argv)
+    convs = dict(args.col)
+    with open(args.csv, newline="", encoding="utf-8") as fh:
+        reader = csv.DictReader(fh)
+        fields = list(reader.fieldnames or [])
+        rows = list(reader)
+    missing = [c for c in convs if c not in fields]
+    if missing:
+        print(f"error: columns not in file: {missing}", file=sys.stderr)
+        return 2
+    data = {c: np.array([_num(r[c]) for r in rows]) for c in convs}
+    try:
+        fit = detect_rows(data, convs, min_delta_bic=args.min_delta_bic)
+    except ValueError as exc:
+        print(f"error: {exc}", file=sys.stderr)
+        return 2
+    print(fit.summary())
+    if args.out:
+        fixed = repair(data, fit, target=args.target)
+        for i, r in enumerate(rows):
+            for c in convs:
+                if np.isfinite(fixed[c][i]):
+                    r[c] = repr(float(fixed[c][i]))
+        with open(args.out, "w", newline="", encoding="utf-8") as fh:
+            w = csv.DictWriter(fh, fieldnames=fields)
+            w.writeheader()
+            w.writerows(rows)
+        print(f"wrote repaired data to {args.out}")
+    return 1 if fit.detected else 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())