mlobs 0.1.0__tar.gz

mlobs-0.1.0/.gitignore

@@ -0,0 +1,43 @@
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+
+# Distribution / packaging
+dist/
+build/
+*.egg-info/
+.eggs/
+*.egg
+
+# Testing
+.pytest_cache/
+.coverage
+coverage.xml
+htmlcov/
+.tox/
+
+# Type checking
+.mypy_cache/
+
+# Linting
+.ruff_cache/
+
+# Virtual environments
+.venv/
+venv/
+env/
+
+# Editors
+.vscode/
+.idea/
+*.swp
+*.swo
+
+# Environment files
+.env
+.env.*
+!.env.example
+
+# macOS
+.DS_Store

mlobs-0.1.0/CHANGELOG.md

@@ -0,0 +1,29 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.1.0] - 2026-02-24
+
+### Added
+- `NumericDriftDetector` — Kolmogorov-Smirnov two-sample test for numeric columns
+- `CategoricalDriftDetector` — Pearson chi-squared test for categorical columns
+- `PSIDriftDetector` — Population Stability Index with quantile-based binning
+- `JSDriftDetector` — Jensen-Shannon Divergence detector
+- `DriftReport` and `ColumnDriftResult` dataclasses with `to_dict` / `from_dict`
+  serialisation
+- `PipelineLogger` — records per-column feature statistics at named pipeline steps
+- `JSONFormatter` — handles numpy scalars, arrays, NaN → null
+- Backend adapters for **pandas**, **polars** (DataFrame + LazyFrame), and
+  **pyarrow** (Table)
+- `get_adapter()` factory with lazy imports for each backend
+- `DataFrameAdapter` Protocol (PEP 544, runtime-checkable)
+- PEP 561 `py.typed` marker for inline type annotations
+- GitHub Actions CI matrix (Python 3.9–3.12) and PyPI publish workflow
+
+[Unreleased]: https://github.com/your-org/mlobs/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/your-org/mlobs/releases/tag/v0.1.0

mlobs-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 mlobs contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

mlobs-0.1.0/PKG-INFO

@@ -0,0 +1,319 @@
+Metadata-Version: 2.4
+Name: mlobs
+Version: 0.1.0
+Summary: ML observability for structured dataframes: drift detection and pipeline logging
+Project-URL: Homepage, https://github.com/Ahalya24/mlobs
+Project-URL: Repository, https://github.com/Ahalya24/mlobs
+Project-URL: Bug Tracker, https://github.com/Ahalya24/mlobs/issues
+Project-URL: Changelog, https://github.com/Ahalya24/mlobs/blob/main/CHANGELOG.md
+License: MIT License
+        
+        Copyright (c) 2026 mlobs contributors
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+License-File: LICENSE
+Keywords: dataframes,drift,machine-learning,mlops,monitoring,observability
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Requires-Dist: numpy>=1.21
+Requires-Dist: scipy>=1.7
+Provides-Extra: all
+Requires-Dist: pandas>=1.3; extra == 'all'
+Requires-Dist: polars>=0.18; extra == 'all'
+Requires-Dist: pyarrow>=10.0; extra == 'all'
+Provides-Extra: arrow
+Requires-Dist: pyarrow>=10.0; extra == 'arrow'
+Provides-Extra: dev
+Requires-Dist: mypy>=1.5; extra == 'dev'
+Requires-Dist: pandas>=1.3; extra == 'dev'
+Requires-Dist: polars>=0.18; extra == 'dev'
+Requires-Dist: pyarrow>=10.0; extra == 'dev'
+Requires-Dist: pytest-cov>=4.1; extra == 'dev'
+Requires-Dist: pytest>=7.4; extra == 'dev'
+Requires-Dist: ruff>=0.1; extra == 'dev'
+Provides-Extra: pandas
+Requires-Dist: pandas>=1.3; extra == 'pandas'
+Provides-Extra: polars
+Requires-Dist: polars>=0.18; extra == 'polars'
+Description-Content-Type: text/markdown
+
+# mlobs
+
+**ML observability for structured dataframes.**
+
+`mlobs` provides drift detection and pipeline step logging for pandas, polars,
+and PyArrow dataframes — with no dependency on any external ML platform.
+
+[![CI](https://github.com/your-org/mlobs/actions/workflows/ci.yml/badge.svg)](https://github.com/your-org/mlobs/actions/workflows/ci.yml)
+[![PyPI version](https://img.shields.io/pypi/v/mlobs.svg)](https://pypi.org/project/mlobs/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+[![Python 3.9+](https://img.shields.io/pypi/pyversions/mlobs.svg)](https://pypi.org/project/mlobs/)
+
+---
+
+## Installation
+
+```bash
+# Core (numpy + scipy — no backend adapters)
+pip install mlobs
+
+# With a specific backend
+pip install "mlobs[pandas]"
+pip install "mlobs[polars]"
+pip install "mlobs[arrow]"
+
+# All backends
+pip install "mlobs[all]"
+
+# All backends + dev tools (pytest, mypy, ruff)
+pip install "mlobs[dev]"
+```
+
+---
+
+## Quick Start
+
+### Drift Detection
+
+```python
+import pandas as pd
+import numpy as np
+from mlobs import NumericDriftDetector, CategoricalDriftDetector, DriftReport
+from mlobs import get_adapter
+
+# Load reference (training) and current (production) data
+ref = pd.DataFrame({
+    "age":      np.random.default_rng(0).normal(30, 5, 1000),
+    "income":   np.random.default_rng(1).normal(50000, 10000, 1000),
+    "segment":  np.random.default_rng(2).choice(["A", "B", "C"], 1000),
+})
+cur = pd.DataFrame({
+    "age":      np.random.default_rng(0).normal(35, 5, 1000),   # shifted
+    "income":   np.random.default_rng(1).normal(50000, 10000, 1000),
+    "segment":  np.random.default_rng(3).choice(["B", "C", "D"], 1000),  # shifted
+})
+
+adapter = get_adapter(ref)
+
+# Test each column with the appropriate detector
+ks  = NumericDriftDetector(p_value_threshold=0.05)
+chi = CategoricalDriftDetector(p_value_threshold=0.05)
+
+results = [
+    ks.detect(adapter.to_numpy(ref, "age"),
+              adapter.to_numpy(cur, "age"),
+              column_name="age"),
+    ks.detect(adapter.to_numpy(ref, "income"),
+              adapter.to_numpy(cur, "income"),
+              column_name="income"),
+    chi.detect(adapter.to_numpy(ref, "segment"),
+               adapter.to_numpy(cur, "segment"),
+               column_name="segment"),
+]
+
+report = DriftReport(
+    reference_name="2024-Q1",
+    current_name="2024-Q2",
+    results=results,
+)
+
+print(report.summary)
+# {'total_columns': 3, 'drifted': 2, 'not_drifted': 1}
+
+print(report.drifted_columns)
+# ['age', 'segment']
+
+print(report.to_json())
+```
+
+### Pipeline Logging
+
+```python
+import pandas as pd
+from mlobs import PipelineLogger
+
+logger = PipelineLogger(name="preprocessing")
+
+raw = pd.read_csv("data.csv")
+logger.log_step(raw, step_name="raw_input", metadata={"source": "data.csv"})
+
+cleaned = raw.dropna()
+logger.log_step(cleaned, step_name="after_drop_na")
+
+scaled = cleaned.copy()
+scaled["age"] = (scaled["age"] - scaled["age"].mean()) / scaled["age"].std()
+logger.log_step(scaled, step_name="after_scaling", columns=["age"])
+
+# Persist the log as JSON
+logger.dump("pipeline_log.json")
+
+# Or get as a Python dict / JSON string
+d = logger.to_dict()
+json_str = logger.to_json()
+```
+
+### Context Manager
+
+```python
+from mlobs import PipelineLogger
+
+with PipelineLogger(name="my_pipeline") as logger:
+    logger.log_step(df1, "step_1")
+    logger.log_step(df2, "step_2")
+
+print(logger.records)
+```
+
+---
+
+## Supported Drift Tests
+
+| Detector | Column type | Test | p-value |
+|---|---|---|---|
+| `NumericDriftDetector` | numeric | Kolmogorov-Smirnov two-sample | yes |
+| `CategoricalDriftDetector` | categorical | Pearson chi-squared | yes |
+| `PSIDriftDetector` | numeric / categorical | Population Stability Index | no |
+| `JSDriftDetector` | numeric / categorical | Jensen-Shannon Divergence | no |
+
+**PSI thresholds** (conventional):
+- PSI < 0.10 → no significant change
+- 0.10 ≤ PSI < 0.20 → moderate change
+- PSI ≥ 0.20 → significant drift
+
+---
+
+## Multi-Backend Support
+
+The same API works across all supported backends:
+
+```python
+import polars as pl
+import pyarrow as pa
+from mlobs import get_adapter, NumericDriftDetector
+
+# polars
+df_pl = pl.read_parquet("data.parquet")
+adapter = get_adapter(df_pl)
+arr = adapter.to_numpy(df_pl, "age")
+
+# pyarrow
+table = pa.ipc.open_file("data.arrow").read_all()
+adapter = get_adapter(table)
+arr = adapter.to_numpy(table, "age")
+```
+
+---
+
+## API Reference
+
+### Drift Detection
+
+```python
+from mlobs import (
+    NumericDriftDetector,
+    CategoricalDriftDetector,
+    PSIDriftDetector,
+    JSDriftDetector,
+    DriftReport,
+    ColumnDriftResult,
+)
+```
+
+**`NumericDriftDetector(p_value_threshold=0.05, alternative="two-sided")`**
+- `.detect(reference, current, column_name="unknown") -> ColumnDriftResult`
+
+**`CategoricalDriftDetector(p_value_threshold=0.05, min_expected_count=5.0)`**
+- `.detect(reference, current, column_name="unknown") -> ColumnDriftResult`
+
+**`PSIDriftDetector(threshold=0.20, n_bins=10, epsilon=1e-4)`**
+- `.detect(reference, current, column_name="unknown", is_categorical=False) -> ColumnDriftResult`
+
+**`JSDriftDetector(threshold=0.1, n_bins=10, epsilon=1e-4)`**
+- `.detect(reference, current, column_name="unknown", is_categorical=False) -> ColumnDriftResult`
+
+**`DriftReport`**
+- `.summary -> dict` — `{total_columns, drifted, not_drifted}`
+- `.drifted_columns -> list[str]`
+- `.to_dict() -> dict`
+- `.to_json(indent=2) -> str`
+- `.from_dict(d) -> DriftReport` (classmethod)
+
+### Pipeline Logging
+
+```python
+from mlobs import PipelineLogger, StepRecord, ColumnStats
+```
+
+**`PipelineLogger(name="pipeline")`**
+- `.log_step(df, step_name, columns=None, metadata=None) -> StepRecord`
+- `.records -> list[StepRecord]`
+- `.clear()`
+- `.to_dict() -> dict`
+- `.to_json(indent=2) -> str`
+- `.dump(path)`
+
+### Adapters
+
+```python
+from mlobs import get_adapter, DataFrameAdapter
+```
+
+**`get_adapter(df) -> DataFrameAdapter`** — auto-detects backend
+
+**`DataFrameAdapter`** Protocol methods (all take `df` as first arg):
+- `shape(df) -> (n_rows, n_cols)`
+- `column_names(df) -> list[str]`
+- `is_numeric(df, col) -> bool`
+- `to_numpy(df, col) -> np.ndarray`
+- `compute_column_stats(df, col) -> ColumnStats`
+
+---
+
+## Contributing
+
+```bash
+git clone https://github.com/your-org/mlobs.git
+cd mlobs
+pip install -e ".[dev]"
+pytest
+```
+
+Run linting and type checking:
+
+```bash
+ruff check src/ tests/
+mypy src/mlobs
+```
+
+---
+
+## License
+
+MIT — see [LICENSE](LICENSE).

mlobs-0.1.0/README.md

@@ -0,0 +1,253 @@
+# mlobs
+
+**ML observability for structured dataframes.**
+
+`mlobs` provides drift detection and pipeline step logging for pandas, polars,
+and PyArrow dataframes — with no dependency on any external ML platform.
+
+[![CI](https://github.com/your-org/mlobs/actions/workflows/ci.yml/badge.svg)](https://github.com/your-org/mlobs/actions/workflows/ci.yml)
+[![PyPI version](https://img.shields.io/pypi/v/mlobs.svg)](https://pypi.org/project/mlobs/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+[![Python 3.9+](https://img.shields.io/pypi/pyversions/mlobs.svg)](https://pypi.org/project/mlobs/)
+
+---
+
+## Installation
+
+```bash
+# Core (numpy + scipy — no backend adapters)
+pip install mlobs
+
+# With a specific backend
+pip install "mlobs[pandas]"
+pip install "mlobs[polars]"
+pip install "mlobs[arrow]"
+
+# All backends
+pip install "mlobs[all]"
+
+# All backends + dev tools (pytest, mypy, ruff)
+pip install "mlobs[dev]"
+```
+
+---
+
+## Quick Start
+
+### Drift Detection
+
+```python
+import pandas as pd
+import numpy as np
+from mlobs import NumericDriftDetector, CategoricalDriftDetector, DriftReport
+from mlobs import get_adapter
+
+# Load reference (training) and current (production) data
+ref = pd.DataFrame({
+    "age":      np.random.default_rng(0).normal(30, 5, 1000),
+    "income":   np.random.default_rng(1).normal(50000, 10000, 1000),
+    "segment":  np.random.default_rng(2).choice(["A", "B", "C"], 1000),
+})
+cur = pd.DataFrame({
+    "age":      np.random.default_rng(0).normal(35, 5, 1000),   # shifted
+    "income":   np.random.default_rng(1).normal(50000, 10000, 1000),
+    "segment":  np.random.default_rng(3).choice(["B", "C", "D"], 1000),  # shifted
+})
+
+adapter = get_adapter(ref)
+
+# Test each column with the appropriate detector
+ks  = NumericDriftDetector(p_value_threshold=0.05)
+chi = CategoricalDriftDetector(p_value_threshold=0.05)
+
+results = [
+    ks.detect(adapter.to_numpy(ref, "age"),
+              adapter.to_numpy(cur, "age"),
+              column_name="age"),
+    ks.detect(adapter.to_numpy(ref, "income"),
+              adapter.to_numpy(cur, "income"),
+              column_name="income"),
+    chi.detect(adapter.to_numpy(ref, "segment"),
+               adapter.to_numpy(cur, "segment"),
+               column_name="segment"),
+]
+
+report = DriftReport(
+    reference_name="2024-Q1",
+    current_name="2024-Q2",
+    results=results,
+)
+
+print(report.summary)
+# {'total_columns': 3, 'drifted': 2, 'not_drifted': 1}
+
+print(report.drifted_columns)
+# ['age', 'segment']
+
+print(report.to_json())
+```
+
+### Pipeline Logging
+
+```python
+import pandas as pd
+from mlobs import PipelineLogger
+
+logger = PipelineLogger(name="preprocessing")
+
+raw = pd.read_csv("data.csv")
+logger.log_step(raw, step_name="raw_input", metadata={"source": "data.csv"})
+
+cleaned = raw.dropna()
+logger.log_step(cleaned, step_name="after_drop_na")
+
+scaled = cleaned.copy()
+scaled["age"] = (scaled["age"] - scaled["age"].mean()) / scaled["age"].std()
+logger.log_step(scaled, step_name="after_scaling", columns=["age"])
+
+# Persist the log as JSON
+logger.dump("pipeline_log.json")
+
+# Or get as a Python dict / JSON string
+d = logger.to_dict()
+json_str = logger.to_json()
+```
+
+### Context Manager
+
+```python
+from mlobs import PipelineLogger
+
+with PipelineLogger(name="my_pipeline") as logger:
+    logger.log_step(df1, "step_1")
+    logger.log_step(df2, "step_2")
+
+print(logger.records)
+```
+
+---
+
+## Supported Drift Tests
+
+| Detector | Column type | Test | p-value |
+|---|---|---|---|
+| `NumericDriftDetector` | numeric | Kolmogorov-Smirnov two-sample | yes |
+| `CategoricalDriftDetector` | categorical | Pearson chi-squared | yes |
+| `PSIDriftDetector` | numeric / categorical | Population Stability Index | no |
+| `JSDriftDetector` | numeric / categorical | Jensen-Shannon Divergence | no |
+
+**PSI thresholds** (conventional):
+- PSI < 0.10 → no significant change
+- 0.10 ≤ PSI < 0.20 → moderate change
+- PSI ≥ 0.20 → significant drift
+
+---
+
+## Multi-Backend Support
+
+The same API works across all supported backends:
+
+```python
+import polars as pl
+import pyarrow as pa
+from mlobs import get_adapter, NumericDriftDetector
+
+# polars
+df_pl = pl.read_parquet("data.parquet")
+adapter = get_adapter(df_pl)
+arr = adapter.to_numpy(df_pl, "age")
+
+# pyarrow
+table = pa.ipc.open_file("data.arrow").read_all()
+adapter = get_adapter(table)
+arr = adapter.to_numpy(table, "age")
+```
+
+---
+
+## API Reference
+
+### Drift Detection
+
+```python
+from mlobs import (
+    NumericDriftDetector,
+    CategoricalDriftDetector,
+    PSIDriftDetector,
+    JSDriftDetector,
+    DriftReport,
+    ColumnDriftResult,
+)
+```
+
+**`NumericDriftDetector(p_value_threshold=0.05, alternative="two-sided")`**
+- `.detect(reference, current, column_name="unknown") -> ColumnDriftResult`
+
+**`CategoricalDriftDetector(p_value_threshold=0.05, min_expected_count=5.0)`**
+- `.detect(reference, current, column_name="unknown") -> ColumnDriftResult`
+
+**`PSIDriftDetector(threshold=0.20, n_bins=10, epsilon=1e-4)`**
+- `.detect(reference, current, column_name="unknown", is_categorical=False) -> ColumnDriftResult`
+
+**`JSDriftDetector(threshold=0.1, n_bins=10, epsilon=1e-4)`**
+- `.detect(reference, current, column_name="unknown", is_categorical=False) -> ColumnDriftResult`
+
+**`DriftReport`**
+- `.summary -> dict` — `{total_columns, drifted, not_drifted}`
+- `.drifted_columns -> list[str]`
+- `.to_dict() -> dict`
+- `.to_json(indent=2) -> str`
+- `.from_dict(d) -> DriftReport` (classmethod)
+
+### Pipeline Logging
+
+```python
+from mlobs import PipelineLogger, StepRecord, ColumnStats
+```
+
+**`PipelineLogger(name="pipeline")`**
+- `.log_step(df, step_name, columns=None, metadata=None) -> StepRecord`
+- `.records -> list[StepRecord]`
+- `.clear()`
+- `.to_dict() -> dict`
+- `.to_json(indent=2) -> str`
+- `.dump(path)`
+
+### Adapters
+
+```python
+from mlobs import get_adapter, DataFrameAdapter
+```
+
+**`get_adapter(df) -> DataFrameAdapter`** — auto-detects backend
+
+**`DataFrameAdapter`** Protocol methods (all take `df` as first arg):
+- `shape(df) -> (n_rows, n_cols)`
+- `column_names(df) -> list[str]`
+- `is_numeric(df, col) -> bool`
+- `to_numpy(df, col) -> np.ndarray`
+- `compute_column_stats(df, col) -> ColumnStats`
+
+---
+
+## Contributing
+
+```bash
+git clone https://github.com/your-org/mlobs.git
+cd mlobs
+pip install -e ".[dev]"
+pytest
+```
+
+Run linting and type checking:
+
+```bash
+ruff check src/ tests/
+mypy src/mlobs
+```
+
+---
+
+## License
+
+MIT — see [LICENSE](LICENSE).